Page MenuHomePhabricator
Create Task
Maniphest T196604

Phabricator replica functionality broken, needs port configuration
Closed, ResolvedPublic
Actions

Description

The master database for phabricator has been failed over to a larger, newer host (we almost doubled the memory).

However, it is know some functionality happen to break on the phabricator replicas. As the master has been switched, so has the replica. The DNS has been updated accordingly. However- the new replica for phabricator runs on port 3323, not the standard one 3306. We warned some weeks in advance, misc services had to be prepared for different port configuration, and in the case of phabricator, a new separate option has to be puppetized (mysql.port), and I would recommend exposing it at

hieradata/role/*/phabricator.yaml:phabricator::mysql::slave::port

For example (details are left to the phabricator owners). There are plans to change the master port at some point, so I recommend fixing the master configuration, too, even if we will leave it on port default/3306 at the moment

Details

SubjectRepoBranchLines +/-
phabricator weekly project changes email: Add mysql slave port parameteroperations/puppetproduction+24 -22
phabricator: Add mysql slave port parameter and provide value via Hieraoperations/puppetproduction+16 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Jun 7 2018, 6:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2018, 6:40 AM
jcrespo added a parent task: T132324: Tracking and Reducing cron-spam to root@ .Jun 7 2018, 1:06 PM
jcrespo added a comment.Jun 7 2018, 1:13 PM

Also seen as:

Traceback (most recent call last):
  File "/srv/phab/tools/public_task_dump.py", line 25, in <module>
    mdb = dbcon('phabricator_maniphest')
  File "/srv/phab/tools/public_task_dump.py", line 23, in dbcon
    passwd=c.phuser_passwd)
  File "/srv/deployment/phabricator/deployment/tools/wmfphablib/phabdb.py", line 1464, in __init__
    self.conn = self.connect()
  File "/srv/deployment/phabricator/deployment/tools/wmfphablib/phabdb.py", line 1471, in connect
    charset=self.charset)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'm3-slave.eqiad.wmnet' (110)")
gerritbot subscribed.Jun 8 2018, 12:15 AM

Change 438128 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Add a mysql slave port parameter and provide it's value via hiera

https://gerrit.wikimedia.org/r/438128

gerritbot added a project: Patch-For-Review.Jun 8 2018, 12:15 AM
• mmodell mentioned this in rPHDEPb84494c0f0cd: mysql slave moved to an alternate port.Jun 8 2018, 12:23 AM
gerritbot added a comment.Jun 8 2018, 6:36 AM

Change 438128 merged by Dzahn:
[operations/puppet@production] phabricator: Add mysql slave port parameter and provide value via Hiera

https://gerrit.wikimedia.org/r/438128

jcrespo updated the task description. (Show Details)Jun 8 2018, 9:23 AM
• mmodell added a comment.Jun 8 2018, 12:23 PM

This should be resolved now. Will leave open until I confirm that the dump is updated.

• mmodell mentioned this in rPHDEP2c10165af03b: Update tools submodule.Jun 8 2018, 12:28 PM
jcrespo awarded a token.Jun 8 2018, 12:32 PM
• mmodell closed this task as Resolved.Jun 8 2018, 8:13 PM
• mmodell claimed this task.
twentyafterfour@phab1001:~$ ls /srv/dumps/ -la
total 1050732
drwxr-xr-x 2 root root       4096 Aug  8  2017 .
drwxr-xr-x 8 root root       4096 Feb  5 15:22 ..
-rw-r--r-- 1 root root 1075934511 Jun  8 12:58 phabricator_public.dump
twentyafterfour@phab1001:~$
gerritbot added a comment.Jun 11 2018, 11:30 AM

Change 439581 had a related patch set uploaded (by Aklapper; owner: Aklapper):
[operations/puppet@production] phabricator weekly project changes email: Add mysql slave port parameter

https://gerrit.wikimedia.org/r/439581

gerritbot added a comment.Jun 12 2018, 7:13 AM

Change 439581 merged by Dzahn:
[operations/puppet@production] phabricator weekly project changes email: Add mysql slave port parameter

https://gerrit.wikimedia.org/r/439581

• Vvjjkkii renamed this task from Phabricator replica functionality broken, needs port configuration to 3hbaaaaaaa.Jul 1 2018, 1:06 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed • mmodell as the assignee of this task.
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
Marostegui renamed this task from 3hbaaaaaaa to Phabricator replica functionality broken, needs port configuration.Jul 2 2018, 5:04 AM
Marostegui closed this task as Resolved.
Marostegui assigned this task to • mmodell.
Marostegui lowered the priority of this task from High to Medium.
Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
Marostegui updated the task description. (Show Details)
CommunityTechBot raised the priority of this task from Medium to Needs Triage.Jul 5 2018, 6:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL