Phabricator just got improved clustering support. See upstream [[ https://secure.phabricator.com/w/changelog/2016.16/ | changelog/2016.16 ]] for details.
Key details from the [[ https://secure.phabricator.com/book/phabricator/article/cluster_databases/ | phabricator database cluster documentation ]]:
>If you lose the master, Phabricator can degrade automatically into read-only mode and remain available, but can not fully recover without operational intervention unless the master recovers on its own.
>
>Phabricator will not currently send read traffic to replicas unless the master has failed, so configuring a replica will not currently spread any load away from the master. Future versions of Phabricator are expected to be able to distribute some read traffic to replicas.
>
>Phabricator can not currently be configured into a multi-master mode, nor can it be configured to automatically promote a replica to become the new master. There are no current plans to support multi-master mode or autonomous failover, although this may change in the future.
----
=== Old Description:
Following up from T109279, we need to keep an eye on phabricator performance and scalability concerns.
Some changes may be necessary in the coming months to ensure that we are adequately prepared to keep phabricator performance at an acceptable level as we continue to rely on phabricator for more critical functions.
There are a few areas where phabricator can be configured for better scalability:
# File storage.
- Currently we have phabricator configured to use the mysql storage engine for file attachments and temporary storage. This stores files directly in a blob column within phabricator's mysql cluster. I'm not aware of any serious performance issues with this method, however, it does add a lot of extra load to the mysql cluster and requires extra database connections from the front-end web servers to the mysql servers.
- An alternative option would be to move phabricator storage to openstack swift. This can probably be accomplished fairly easily by customizing the amazon s3 engine to work with openstack. This should be trivial, as far as I can tell. The major concern here is that we do not currently use swift for storing sensitive information. Unfortunately, some sensitive information is uploaded to phabricator and whatever storage system we use needs to protect these sensitive files from accidental disclosure.
# Repository hosting
* As we are migrating repositories from gerrit to phabricator, a significant amount of load will be placed on phabricator by requests to `git` service. Both CPU and Disk I/O could become bottlenecks which could potentially slow developer activity and limit CI testing throughput. The upstream phabricator project seems to have a plan for addressing scalability of SCM hosting, which primarily revolves around implementing a smart protocol proxy in phabricator which can delegate the work to multiple back-end git servers. With repositories distributed across several git servers, performance and scaling should not be a problem. This is, however, currently unimplemented and it could be a while before the upstream gets this working.