Page MenuHomePhabricator
Create Task
Maniphest T195595

db1061 (s6 primary master) has a wrong live server_id - needs a MySQL restart
Closed, ResolvedPublic
Actions

Description

We have found an issue with db1061's server_id

Right now db1061 has a different server_id on mysql than the one on the config file.

root@db1061:~# cat /etc/my.cnf | grep server_id

server_id  = 171974883

root@db1061:~# mysql --skip-ssl -e "select @@server_id"

+-------------+

| @@server_id |

+-------------+

|   171978766 |

+-------------+

The live server_id config was generated from when db1061 had the IP: 10.64.48.14 (https://gerrit.wikimedia.org/r/#/c/117358/2/templates/wmnet)

However, at somepoint it was probably moved and got the new IP it has now, which generates the server_id value that is on my.cnf but probably puppet wasn't run before starting mysql and hence it got: 171978766

The problem comes when db1125 (sanitarium) has 10.64.48.14 and thus 171978766 as server_id, when it tries to replicate from db1061 it cannot write, as events with the same server_id are ignored.

So during the read_only window for T187962 we should try to:

  • Stop MySQL
  • apt full-upgrade
  • Run puppet (so the new socket is picked up)
  • Start MySQL
  • Run mysql_upgrade (can be done anytime, no need to do it during the maintenance window)

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
sanitarium_multi: Hardcode db1125 server_idMarosteguioperations/puppetproduction+4 -0
db1061: Upgrade socket locationMarosteguioperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.May 25 2018, 3:11 PM
Herald added a subscriber: Aklapper. · View Herald TranscriptMay 25 2018, 3:11 PM
Marostegui triaged this task as Medium priority.May 25 2018, 3:12 PM
Marostegui moved this task from Triage to Pending comment on the DBA board.
gerritbot subscribed.May 25 2018, 3:27 PM

Change 435182 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] sanitarium_multi: Hardcode db1125 server_id

https://gerrit.wikimedia.org/r/435182

gerritbot added a project: Patch-For-Review.May 25 2018, 3:27 PM
Marostegui updated the task description. (Show Details)May 25 2018, 3:31 PM
gerritbot added a comment.May 25 2018, 3:32 PM

Change 435182 merged by Marostegui:
[operations/puppet@production] sanitarium_multi: Hardcode db1125 server_id

https://gerrit.wikimedia.org/r/435182

Marostegui updated the task description. (Show Details)May 25 2018, 3:36 PM
• jcrespo mentioned this in T195598: Differing database server ips and server_id numbers.May 25 2018, 3:46 PM
• jcrespo added a parent task: T195598: Differing database server ips and server_id numbers.
Marostegui moved this task from Pending comment to In progress on the DBA board.May 27 2018, 7:48 AM
gerritbot added a comment.May 28 2018, 9:49 AM

Change 435757 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1061: Upgrade socket location

https://gerrit.wikimedia.org/r/435757

Stashbot subscribed.May 29 2018, 9:13 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-29T09:13:00Z] <marostegui> Downtime s6 replicas for 4 hours - T195595

gerritbot added a comment.May 29 2018, 11:02 AM

Change 435757 merged by Marostegui:
[operations/puppet@production] db1061: Upgrade socket location

https://gerrit.wikimedia.org/r/435757

Marostegui added a comment.May 29 2018, 11:14 AM

This restart has been done, current values:

+------------+
| @@hostname |
+------------+
| db1061     |
+------------+
1 row in set (0.00 sec)

+-------------+
| @@server_id |
+-------------+
|   171974883 |
+-------------+
1 row in set (0.00 sec)

root@db1061:~# cat /etc/my.cnf | grep server_id
server_id  = 171974883
Marostegui added a comment.May 29 2018, 11:15 AM

We can now revert: https://gerrit.wikimedia.org/r/#/c/435182/

Stashbot added a comment.May 29 2018, 11:21 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-29T11:21:15Z] <marostegui> Restar db1125 mysql - T195595

Marostegui closed this task as Resolved.May 29 2018, 11:24 AM
Marostegui claimed this task.

This is all done, including restarting db1125:s6 to pick up the new server_id.

• Vvjjkkii renamed this task from db1061 (s6 primary master) has a wrong live server_id - needs a MySQL restart to 49baaaaaaa.Jul 1 2018, 1:07 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed Marostegui as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
Marostegui renamed this task from 49baaaaaaa to db1061 (s6 primary master) has a wrong live server_id - needs a MySQL restart.Jul 1 2018, 7:54 PM
Marostegui closed this task as Resolved.
Marostegui claimed this task.
Marostegui lowered the priority of this task from High to Medium.
Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
Marostegui updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits