Page MenuHomePhabricator

Phabricator replica functionality broken, needs port configuration
Closed, ResolvedPublic

Description

The master database for phabricator has been failed over to a larger, newer host (we almost doubled the memory).

However, it is know some functionality happen to break on the phabricator replicas. As the master has been switched, so has the replica. The DNS has been updated accordingly. However- the new replica for phabricator runs on port 3323, not the standard one 3306. We warned some weeks in advance, misc services had to be prepared for different port configuration, and in the case of phabricator, a new separate option has to be puppetized (mysql.port), and I would recommend exposing it at

hieradata/role/*/phabricator.yaml:phabricator::mysql::slave::port

For example (details are left to the phabricator owners). There are plans to change the master port at some point, so I recommend fixing the master configuration, too, even if we will leave it on port default/3306 at the moment

Event Timeline

jcrespo created this task.Jun 7 2018, 6:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2018, 6:40 AM

Also seen as:

Traceback (most recent call last):
  File "/srv/phab/tools/public_task_dump.py", line 25, in <module>
    mdb = dbcon('phabricator_maniphest')
  File "/srv/phab/tools/public_task_dump.py", line 23, in dbcon
    passwd=c.phuser_passwd)
  File "/srv/deployment/phabricator/deployment/tools/wmfphablib/phabdb.py", line 1464, in __init__
    self.conn = self.connect()
  File "/srv/deployment/phabricator/deployment/tools/wmfphablib/phabdb.py", line 1471, in connect
    charset=self.charset)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'm3-slave.eqiad.wmnet' (110)")

Change 438128 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Add a mysql slave port parameter and provide it's value via hiera

https://gerrit.wikimedia.org/r/438128

Change 438128 merged by Dzahn:
[operations/puppet@production] phabricator: Add mysql slave port parameter and provide value via Hiera

https://gerrit.wikimedia.org/r/438128

jcrespo updated the task description. (Show Details)Jun 8 2018, 9:23 AM

This should be resolved now. Will leave open until I confirm that the dump is updated.

mmodell closed this task as Resolved.Jun 8 2018, 8:13 PM
mmodell claimed this task.
twentyafterfour@phab1001:~$ ls /srv/dumps/ -la
total 1050732
drwxr-xr-x 2 root root       4096 Aug  8  2017 .
drwxr-xr-x 8 root root       4096 Feb  5 15:22 ..
-rw-r--r-- 1 root root 1075934511 Jun  8 12:58 phabricator_public.dump
twentyafterfour@phab1001:~$

Change 439581 had a related patch set uploaded (by Aklapper; owner: Aklapper):
[operations/puppet@production] phabricator weekly project changes email: Add mysql slave port parameter

https://gerrit.wikimedia.org/r/439581

Change 439581 merged by Dzahn:
[operations/puppet@production] phabricator weekly project changes email: Add mysql slave port parameter

https://gerrit.wikimedia.org/r/439581

Vvjjkkii renamed this task from Phabricator replica functionality broken, needs port configuration to 3hbaaaaaaa.Jul 1 2018, 1:06 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed mmodell as the assignee of this task.
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
Marostegui renamed this task from 3hbaaaaaaa to Phabricator replica functionality broken, needs port configuration.Jul 2 2018, 5:04 AM
Marostegui closed this task as Resolved.
Marostegui assigned this task to mmodell.
Marostegui lowered the priority of this task from High to Normal.
Marostegui updated the task description. (Show Details)
CommunityTechBot raised the priority of this task from Normal to Needs Triage.Jul 5 2018, 6:58 PM