phabricator dump script should use slave db, not master
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dzahn
	Sep 10 2015, 10:06 PM

Description

follow-up to T112135, and because of T109279

make the phabricator dump script use m3-slave instead of m3-master as the database it connects to

Mentioned In: T109279: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections.
T112135: Phabricator dump frozen since Aug 28
Mentioned Here: T109279: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections.
T112135: Phabricator dump frozen since Aug 28

Dzahn raised the priority of this task from to Needs Triage.

Dzahn updated the task description. (Show Details)

So, a couple of clarifications:

The dump process is relatively expensive, it took today 2 hours from 2 to 4 am UTC. It doesn't seem to be consistent, though, so I may want to review that
The dump host is controlled by puppet:modules/phabricator/templates/phabtools.conf.erb, however, I suppose some tools may require access to the actual master (I may need help here to fully understand all the process and make sure we do not break it). If that is true, we need to change the software at: phabricator/tools
The dumps are not the cause of the spike, but they are back (before the dumps were restarted), so allow me to triage this higher than I thought, as "normal" to avoid unnecessary traffic.

jcrespo moved this task from Triage to Backlog on the DBA board.

I do not have time to work on this, but using the master seems to be creating high contention, and could be the cause of new connection errors.

I should have blocked from allowing you to use the master in the first place.

I did this :)