Page MenuHomePhabricator
Create Task
Maniphest T197069

Failover db1052 (s1) db primary master
Closed, ResolvedPublic
Actions

Description

s1 primary master db1052 needs to be decommissioned (T186320) and it is also blocking row B switch upgrade (T183585)

The candidate master is db1067.

This a checklist of what needs to be done before the failover

  • Verify db1067 has STATEMENT as binlog format
  • Upgrade micro-code and reboot on db1067

or

  • Reimage db1067 to stretch/10.1 and check the latest micro-code is installed and active (?)
  • Pick a date for the failover: July 18th (Wednesday) at 06:00AM UTC
  • Communicate liaisons to handle the read only time T197134
  • Pick and prepare a new candidate master (db1089 - row C, which has already been migrated to the new switch)
  •  Change this to db1083 once db1067 is the master and the network maintenance has been done (T197069#4418823) This step will be followed up at T199861

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 13 2018, 8:22 AM
• Marostegui moved this task from Triage to Backlog on the DBA board.Jun 13 2018, 8:23 AM

I would like to suggest July 18th (Wednesday) at 06:00AM UTC as a failover date

• Marostegui added a parent task: T186320: Decommission db1051-db1060 (DBA tracking).Jun 13 2018, 8:23 AM
• Marostegui moved this task from Backlog to Pending comment on the DBA board.
• Marostegui added a subscriber: ayounsi.
jcrespo added a comment.Jun 13 2018, 8:27 AM

Seems ok to me at first. I would also like to check for blockers for the parent task, even if they are not blockers for this subtask.

• Marostegui updated the task description. (Show Details)Jun 13 2018, 8:28 AM
jcrespo updated the task description. (Show Details)Jun 13 2018, 8:28 AM
jcrespo updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)Jun 13 2018, 8:44 AM
• Marostegui updated the task description. (Show Details)Jun 13 2018, 9:20 AM
• Marostegui mentioned this in T186320: Decommission db1051-db1060 (DBA tracking).Jun 13 2018, 3:35 PM
• Marostegui added a subtask: T197134: Announce 30 minutes read-only time for enwiki 18th July 06:00AM UTC.Jun 13 2018, 3:41 PM
• Marostegui updated the task description. (Show Details)Jun 13 2018, 3:43 PM
• Marostegui mentioned this in T89737: Make several mediawiki table fields unsigned ints on wmf databases.Jun 25 2018, 1:02 PM
• Marostegui mentioned this in T195193: Schema change for ct_tag_id field to change_tag.
• Marostegui mentioned this in T192926: Schema change to drop archive.ar_text and archive.ar_flags.
• Marostegui mentioned this in T191316: Schema change to make archive.ar_rev_id NOT NULL.
gerritbot subscribed.Jun 27 2018, 8:54 AM

Change 442250 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1067 for reimage

https://gerrit.wikimedia.org/r/442250

gerritbot added a project: Patch-For-Review.Jun 27 2018, 8:54 AM
gerritbot added a comment.Jun 27 2018, 9:00 AM

Change 442250 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1067 for reimage

https://gerrit.wikimedia.org/r/442250

gerritbot added a comment.Jun 27 2018, 9:05 AM

Change 442252 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: db1067: disable notifications and reinstall as stretch

https://gerrit.wikimedia.org/r/442252

gerritbot added a comment.Jun 27 2018, 10:06 AM

Change 442252 merged by Jcrespo:
[operations/puppet@production] mariadb: db1067: disable notifications and reinstall as stretch

https://gerrit.wikimedia.org/r/442252

ops-monitoring-bot subscribed.Jun 27 2018, 10:16 AM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db1067.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806271016_jynus_25771.log.

ops-monitoring-bot added a comment.Jun 27 2018, 10:37 AM

Completed auto-reimage of hosts:

['db1067.eqiad.wmnet']

and were ALL successful.

jcrespo updated the task description. (Show Details)Jun 27 2018, 11:21 AM
gerritbot added a comment.Jun 27 2018, 11:40 AM

Change 442279 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Reenable notifications on db1067

https://gerrit.wikimedia.org/r/442279

gerritbot added a comment.Jun 27 2018, 2:37 PM

Change 442279 merged by Jcrespo:
[operations/puppet@production] mariadb: Reenable notifications on db1067

https://gerrit.wikimedia.org/r/442279

• Vvjjkkii renamed this task from Failover db1052 (s1) db primary master to 64aaaaaaaa.Jul 1 2018, 1:04 AM
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
• Marostegui renamed this task from 64aaaaaaaa to Failover db1052 (s1) db primary master.Jul 1 2018, 8:12 PM
• Marostegui lowered the priority of this task from High to Medium.
• Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)Jul 4 2018, 12:32 PM
gerritbot added a comment.Jul 4 2018, 12:33 PM

Change 443825 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/443825

jcrespo added a comment.Jul 4 2018, 12:35 PM

Pick and prepare a new candidate master (db1089 - row C, which has already been migrated to the new switch)

Maybe it is just me, but having as candidate a host that will be on the same row, is it a good idea? Maybe it is a temporary candidate only?

gerritbot added a comment.Jul 4 2018, 12:42 PM

Change 443825 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/443825

Stashbot subscribed.Jul 4 2018, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-04T12:46:23Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 for maintenance - T197069 (duration: 02m 57s)

gerritbot added a comment.Jul 4 2018, 12:51 PM

Change 443826 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1089.yaml: Change binlog format

https://gerrit.wikimedia.org/r/443826

gerritbot added a comment.Jul 4 2018, 12:53 PM

Change 443826 merged by Marostegui:
[operations/puppet@production] db1089.yaml: Change binlog format

https://gerrit.wikimedia.org/r/443826

Stashbot added a comment.Jul 4 2018, 12:56 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-04T12:56:18Z] <marostegui> Stop MySQL and reboot db1089 to upgrade+change it to statement - T197069

• Marostegui updated the task description. (Show Details)Jul 4 2018, 1:17 PM
In T197069#4397353, @jcrespo wrote:

Pick and prepare a new candidate master (db1089 - row C, which has already been migrated to the new switch)

Maybe it is just me, but having as candidate a host that will be on the same row, is it a good idea? Maybe it is a temporary candidate only?

Sorry - I missed this.
I misread where db1067 is, I will replace it with db1083 once db1067 is the master

• Marostegui updated the task description. (Show Details)Jul 4 2018, 1:18 PM
• Marostegui added a comment.Jul 5 2018, 7:55 AM

I am starting with the checklist preparation in the etherpad - I will also start with the patches soon.

• Marostegui mentioned this in T146591: Add a primary key to l10n_cache.Jul 9 2018, 6:10 AM
• Marostegui mentioned this in T196379: Schema change: Add unique index on archive.ar_rev_id.
• Marostegui mentioned this in T197891: Schema change to drop default from externallinks.el_index_60.
jcrespo added a subtask: T199224: Test database master switchover script on codfw.Jul 10 2018, 1:35 PM
• Marostegui updated the task description. (Show Details)Jul 12 2018, 7:06 AM
gerritbot added a comment.Jul 12 2018, 7:08 AM

Change 445349 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1083.yaml: Change binlog format

https://gerrit.wikimedia.org/r/445349

gerritbot added a comment.Jul 12 2018, 7:10 AM

Change 445349 merged by Marostegui:
[operations/puppet@production] db1083.yaml: Change binlog format

https://gerrit.wikimedia.org/r/445349

gerritbot added a comment.Jul 12 2018, 7:12 AM

Change 445350 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/445350

gerritbot added a comment.Jul 12 2018, 7:16 AM

Change 445350 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/445350

• Marostegui added a comment.Jul 12 2018, 7:27 AM

So, I have restarted db1083 with binlog format = STATEMENT.
This host is ready to be the candidate master once db1067 is the new master.

Right now in s1 we have two candidate masters

db1089 -> row C (same row as db1067)
db1083 -> row B (different row as db1067, but this row still needs the switch maintenance, so let's maintain db1089 as candidate until the switch maintenance is done - once done, we can revert db1089 to ROW format so we can have master and candidate in different rows).

• Marostegui updated the task description. (Show Details)Jul 12 2018, 7:27 AM
gerritbot added a comment.Jul 12 2018, 7:29 AM

Change 445352 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1083 with low weight

https://gerrit.wikimedia.org/r/445352

gerritbot added a comment.Jul 12 2018, 7:35 AM

Change 445352 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1083 with low weight

https://gerrit.wikimedia.org/r/445352

gerritbot added a comment.Jul 12 2018, 7:41 AM

Change 445354 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1067 to s1 masters

https://gerrit.wikimedia.org/r/445354

Johan closed subtask T197134: Announce 30 minutes read-only time for enwiki 18th July 06:00AM UTC as Resolved.Jul 12 2018, 7:48 AM
gerritbot added a comment.Jul 12 2018, 8:15 AM

Change 445363 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s1-master alias

https://gerrit.wikimedia.org/r/445363

gerritbot added a comment.Jul 12 2018, 9:06 AM

Change 445369 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Set up s1 on read only

https://gerrit.wikimedia.org/r/445369

gerritbot added a comment.Jul 12 2018, 9:10 AM

Change 445371 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Promote db1067 to master

https://gerrit.wikimedia.org/r/445371

Stashbot added a comment.Jul 16 2018, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-16T14:52:31Z] <marostegui> Change expire_log_days on db1067 - https://phabricator.wikimedia.org/T197069

• Marostegui claimed this task.Jul 16 2018, 3:44 PM
• Marostegui moved this task from Pending comment to In progress on the DBA board.
• Marostegui mentioned this in T183585: Rack/cable/configure asw2-b-eqiad switch stack.
Stashbot added a comment.Jul 18 2018, 4:56 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-18T04:56:32Z] <marostegui> Starting s1 failover pre steps - https://phabricator.wikimedia.org/T197069

gerritbot added a comment.Jul 18 2018, 5:10 AM

Change 445354 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1067 to s1 master

https://gerrit.wikimedia.org/r/445354

gerritbot added a comment.Jul 18 2018, 5:50 AM

Change 445369 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Set up s1 on read only

https://gerrit.wikimedia.org/r/445369

Stashbot added a comment.Jul 18 2018, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-18T06:00:20Z] <marostegui> Starting s1 failover from db1052 to db1067 - T197069

Stashbot added a comment.Jul 18 2018, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-18T06:01:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Set s1 on ready only for maintenance T197069 (duration: 01m 08s)

gerritbot added a comment.Jul 18 2018, 6:03 AM

Change 445371 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Promote db1067 to master

https://gerrit.wikimedia.org/r/445371

Stashbot added a comment.Jul 18 2018, 6:04 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-18T06:04:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Set db1067 as master T197069 (duration: 00m 53s)

Stashbot added a comment.Jul 18 2018, 6:07 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-18T06:07:17Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: read only OFF after failover T197069 (duration: 00m 53s)

Stashbot added a comment.Jul 18 2018, 6:08 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-18T06:08:39Z] <marostegui> s1 failover finished T197069

• Marostegui added a comment.Jul 18 2018, 6:15 AM

This was smoothly done.
Read only times:

Start: 06:01:31
Finish: 06:07:17

gerritbot added a comment.Jul 18 2018, 6:24 AM

Change 445363 merged by Marostegui:
[operations/dns@master] wmnet: Update s1-master alias

https://gerrit.wikimedia.org/r/445363

• Marostegui mentioned this in T199861: Decommission db1052.Jul 18 2018, 6:36 AM
• Marostegui updated the task description. (Show Details)Jul 18 2018, 6:41 AM

Resolving this as it has all be done - including the clean up tasks.

• Marostegui closed this task as Resolved.Jul 18 2018, 6:43 AM
jcrespo awarded a token.Jul 18 2018, 6:43 AM
• Marostegui mentioned this in T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW.Jul 19 2018, 2:12 PM
jcrespo closed subtask T199224: Test database master switchover script on codfw as Resolved.Jul 24 2018, 5:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL