Page MenuHomePhabricator

Move db1066 to row A
Closed, ResolvedPublic

Description

db1066 will replace db1054 as s2 master.

First, it needs to be moved to row A (any rack but A2, A3).

@Cmjohnson would A6 work for you?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.May 4 2018, 9:40 AM
Marostegui moved this task from Triage to In progress on the DBA board.

db1066 is now pooled on s2, so it will need a depooling before shutting it down (and probably disk changes).

Marostegui raised the priority of this task from Medium to High.May 17 2018, 8:47 AM

Given that db1054 (the primary master is having BBU issues), we should move this host asap to the new rack and prepare for a failover.
@Cmjohnson can we do this movement today?

Change 433568 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/433568

Change 433568 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/433568

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:45:30Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1066 for a rack change - T193847 (duration: 01m 21s)

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:46:30Z] <marostegui> Stop MySQL on db1066 for a rack change - T193847

Change 433571 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Updating dns db1066

https://gerrit.wikimedia.org/r/433571

Change 433571 merged by Cmjohnson:
[operations/dns@master] Updating dns db1066

https://gerrit.wikimedia.org/r/433571

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:50:54Z] <marostegui> Power off db1066 for a rack change - T193847

Change 433572 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db.codfw.php: Change db1066 IP

https://gerrit.wikimedia.org/r/433572

Change 433572 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db.codfw.php: Change db1066 IP

https://gerrit.wikimedia.org/r/433572

Mentioned in SAL (#wikimedia-operations) [2018-05-17T13:58:57Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Change db1066 IP - T193847 (duration: 01m 17s)

Mentioned in SAL (#wikimedia-operations) [2018-05-17T14:03:16Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Change db1066 IP - T193847 (duration: 01m 21s)

This has been moved.
So far no BBU issues or anything related.
I am waiting for MySQL to catch up and the DNS to fully propagate before repooling this host.

Change 433689 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1066

https://gerrit.wikimedia.org/r/433689

Change 433689 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1066

https://gerrit.wikimedia.org/r/433689

Mentioned in SAL (#wikimedia-operations) [2018-05-18T05:25:38Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1066 - T193847 (duration: 01m 22s)

Server repooled
Thanks Chris for getting this done!

Marostegui renamed this task from omdaaaaaaa to Move db1066 to row A.Jul 1 2018, 7:33 AM
Marostegui closed this task as Resolved.
Marostegui assigned this task to Cmjohnson.
Marostegui lowered the priority of this task from High to Medium.
Marostegui updated the task description. (Show Details)
Marostegui added subscribers: Aklapper, GerritBot.