As we are migrating repositories from gerrit to phabricator, a significant amount of load will be placed on phabricator by requests to git service. Both CPU and Disk I/O could become bottlenecks which could potentially slow developer activity and limit CI testing throughput. The upstream phabricator project seems to have a plan for addressing scalability of SCM hosting, which primarily revolves around implementing a smart protocol proxy in phabricator which can delegate the work to multiple back-end git servers. With repositories distributed across several git servers, performance and scaling should not be a problem. This is, however, currently unimplemented and it could be a while before the upstream gets this working.

In addition to scaling repositories over multiple backends (which needs upstream and we want, as you point out), we should also be more aggressively tuning the underlying git gc operations so they run frequently on large repositories and at least occasionally on smaller ones. This helps disk I/O tremendously. This we can do now as well as when upstream scales this out.

I think the root cause of connection spikes is usually misbehaving search indexers, either hitting pages that they shouldn't (excluded in robots.txt) or hitting us too fast (not respecting the index delay in robots.txt)

I'm not exactly sure what we can do other than manually blocking these on a case by case basis.

upstream has recently added support for directing read queries to a read-only mysql slave, as well as failing over to a backup mysql master. That should go a long way to making phabricator more scalable, at least at the db layer.

Phabricator was updated today. Which added support for this.

I've submitted a change to operations/puppet which will enable the phabricator cluster features: https://gerrit.wikimedia.org/r/#/c/300494/

This need good planing to coordinate with our proxy setup and dns configuration to see what happens in case slave or master fails.

I'm not sure how to test it thoroughly because I don't have a replicated db setup to test against. Simply configuring it should be safe, according to the documentation it should behave sanely and transparently with the config change I just submitted.

@jcrespo: According to monitoring replicas we should grant REPLICATION CLIENT privileges to the phabricator user so that phabricator can keep track of replication status. Should I file a task to request the additional sql GRANT?

It is ok, but let's do it with an m3-slave dns entry, which I think it is not there.

It seems it is, although not necesarily pointing to the right place.

@mmodell (and/or anyone related to/interested in phab architecture): let's meet- I am all for supporting any clustering efforts and active-active setup, but we already had existing clustering transparent to the application which we need to handle (either to disable it or to change its loginc). We need to coordinate to set the goals and the failover logic. Maybe the proxy is not longer needed; maybe it is but we need to clarify how? We also have 2 datacenters- we should clarify how we intend to use them.

With the many database nodes; and now "clients" supporting phabricator, we should have clear what is the best way to go. This should be easy, we just need to be on the same page.

Phabricator does not do any active cluster failover - it just monitors the slave status so as described in https://secure.phabricator.com/book/phabricator/article/cluster_databases/ - that article simply describes how to use mysql replication which you are surely more familiar with than I am, but I'm pretty sure it's standard mysql stuff - not really phabricator specific.

@jcrespo: It's not essential for REPLICATION CLIENT to be granted, I believe it will still work without it. The main thing phabricator is able to do is degrade into read-only mode when the master is unavailable.

Which is exactly why I want to talk to you, to answer things such as phab2001 role and would it work if master and slave were the same server? Ping me on IRC when you have the time.

There is no issue with the request you mentioned, I just want to understand the best way to proceed with it.

To give you an idea- if the master fails, we currently failover to the slave (but the slave will not have replication working, then, or if it does, it will be meaningless). Maybe it is not worth having a proxy anymore? Hit me on IRC or we can discuss options by email, whatever is more confortable for you. I want to explain you the options and how to combine them with phabricator clustering features.

@jcrespo: Thanks for further explanation. I'll hit you up on IRC.

fwiw "phab1001.eqiad.wmnet" exists now as a CNAME for iridium as suggested by Jaime

Database clustering is already handled by haproxy. There may be some advantages to the built in clustering support, however, it's not very urgent or high priority.