Page MenuHomePhabricator
Create Task
Maniphest T193847

Move db1066 to row A
Closed, ResolvedPublic
Actions

Description

db1066 will replace db1054 as s2 master.

First, it needs to be moved to row A (any rack but A2, A3).

@Cmjohnson would A6 work for you?

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
db-eqiad.php: Repool db1066Marosteguioperations/mediawiki-configmaster+1 -1
db-eqiad,db.codfw.php: Change db1066 IPMarosteguioperations/mediawiki-configmaster+3 -3
Updating dns db1066Cmjohnsonoperations/dnsmaster+2 -2
db-eqiad.php: Depool db1066Marosteguioperations/mediawiki-configmaster+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT186320 Decommission db1051-db1060 (DBA tracking)
 ResolvedMarosteguiT194870 Failover s2 primary master
 Resolved• CmjohnsonT193847 Move db1066 to row A

Event Timeline

Marostegui created this task.May 4 2018, 9:40 AM
Restricted Application added a project: SRE. · View Herald TranscriptMay 4 2018, 9:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.May 4 2018, 9:40 AM
Marostegui moved this task from Triage to In progress on the DBA board.
• Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.May 4 2018, 2:57 PM
Marostegui moved this task from In progress to Pending comment on the DBA board.May 5 2018, 2:28 PM
• jcrespo subscribed.May 11 2018, 5:13 PM

db1066 is now pooled on s2, so it will need a depooling before shutting it down (and probably disk changes).

Marostegui raised the priority of this task from Medium to High.May 17 2018, 8:47 AM
Marostegui mentioned this in T194870: Failover s2 primary master.

Given that db1054 (the primary master is having BBU issues), we should move this host asap to the new rack and prepare for a failover.
@Cmjohnson can we do this movement today?

Marostegui added a parent task: T194870: Failover s2 primary master.May 17 2018, 9:00 AM
gerritbot subscribed.May 17 2018, 1:39 PM

Change 433568 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/433568

gerritbot added a project: Patch-For-Review.May 17 2018, 1:39 PM
gerritbot added a comment.May 17 2018, 1:43 PM

Change 433568 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/433568

Stashbot subscribed.May 17 2018, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:45:30Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1066 for a rack change - T193847 (duration: 01m 21s)

Stashbot added a comment.May 17 2018, 1:46 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:46:30Z] <marostegui> Stop MySQL on db1066 for a rack change - T193847

gerritbot added a comment.May 17 2018, 1:48 PM

Change 433571 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Updating dns db1066

https://gerrit.wikimedia.org/r/433571

gerritbot added a comment.May 17 2018, 1:48 PM

Change 433571 merged by Cmjohnson:
[operations/dns@master] Updating dns db1066

https://gerrit.wikimedia.org/r/433571

Stashbot added a comment.May 17 2018, 1:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:50:54Z] <marostegui> Power off db1066 for a rack change - T193847

gerritbot added a comment.May 17 2018, 1:53 PM

Change 433572 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db.codfw.php: Change db1066 IP

https://gerrit.wikimedia.org/r/433572

gerritbot added a comment.May 17 2018, 1:57 PM

Change 433572 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db.codfw.php: Change db1066 IP

https://gerrit.wikimedia.org/r/433572

Stashbot added a comment.May 17 2018, 1:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:58:57Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Change db1066 IP - T193847 (duration: 01m 17s)

Stashbot added a comment.May 17 2018, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-17T14:03:16Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Change db1066 IP - T193847 (duration: 01m 21s)

Marostegui added a comment.May 17 2018, 2:08 PM

This has been moved.
So far no BBU issues or anything related.
I am waiting for MySQL to catch up and the DNS to fully propagate before repooling this host.

• Cmjohnson moved this task from Up next to High Priority Task on the ops-eqiad board.May 17 2018, 2:42 PM
• Cmjohnson moved this task from High Priority Task to Blocked on the ops-eqiad board.
gerritbot added a comment.May 18 2018, 5:20 AM

Change 433689 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1066

https://gerrit.wikimedia.org/r/433689

gerritbot added a comment.May 18 2018, 5:23 AM

Change 433689 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1066

https://gerrit.wikimedia.org/r/433689

Stashbot added a comment.May 18 2018, 5:25 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-18T05:25:38Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1066 - T193847 (duration: 01m 22s)

Marostegui closed this task as Resolved.May 18 2018, 5:26 AM

Server repooled
Thanks Chris for getting this done!

• Vvjjkkii renamed this task from Move db1066 to row A to omdaaaaaaa.Jul 1 2018, 1:13 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed • Cmjohnson as the assignee of this task.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
Marostegui renamed this task from omdaaaaaaa to Move db1066 to row A.Jul 1 2018, 7:33 AM
Marostegui closed this task as Resolved.
Marostegui assigned this task to • Cmjohnson.
Marostegui lowered the priority of this task from High to Medium.
Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
Marostegui updated the task description. (Show Details)
Marostegui added subscribers: Aklapper, GerritBot.
Aklapper edited subscribers, added: gerritbot; removed: GerritBot.May 16 2023, 10:18 AM
Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 10:38 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits