Move db1051 to row B3
Closed, ResolvedPublic

Description

Hey @Cmjohnson

We would need db10151 to be moved to B3 if possible. This is in order to start mitigating the problem with the switch that got rebooted twice in the last two weeks (T155999).

If possible, can you also update DNS for us?
If you happen to know the IP it will get beforehand, we can change it before we poweroff the server so it will boot up with the correct one.
No worries if you don't, we will go thru ILO and get it changed.

Coordinate with me before the move as we need to depool the server.

Thanks!
Manuel.

D 2 is a potentially bad choice, as it will soon become a 10G rack once the T148506 migration is (finally…) done. Any other rack except D 2 & D 7 (new 10G) and D 6-8 (current 10G) is probably a safe bet.

Thanks Faidon - let's go for B3 then?

Marostegui changed the title from "Move db1051 to row D2" to "Move db1051 to row B3".Jan 23 2017, 3:48 PM
Marostegui edited the task description. (Show Details)

Sure, no objections from a networking/DC design perspective :)

Alerts silenced for 24 hours - I will re-enable them once the move is done.

Change 333873 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333873

Change 333873 merged by Marostegui:
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333873

Mentioned in SAL (#wikimedia-operations) [2017-01-24T12:48:33Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1051 - T156004 (duration: 00m 39s)

Change 333898 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333898

Change 333898 merged by jenkins-bot:
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333898

Mentioned in SAL (#wikimedia-operations) [2017-01-24T12:55:01Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1051 - T156004 (duration: 00m 39s)

Mentioned in SAL (#wikimedia-operations) [2017-01-24T12:56:29Z] <marostegui> Shutdown mysql on db1051 for maintenance - T156004

Change 333899 had a related patch set uploaded (by Cmjohnson):
Updating dns for db1051 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333899

Mentioned in SAL (#wikimedia-operations) [2017-01-24T13:00:44Z] <marostegui> Shutdown db1051 for maintenance - T156004

Change 333899 merged by Cmjohnson:
Updating dns for db1051 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333899

Change 333900 had a related patch set uploaded (by Marostegui):
db-codfw,db-eqiad.php: Update db1051 IP

https://gerrit.wikimedia.org/r/333900

Change 333900 merged by jenkins-bot:
db-codfw,db-eqiad.php: Update db1051 IP

https://gerrit.wikimedia.org/r/333900

Mentioned in SAL (#wikimedia-operations) [2017-01-24T13:14:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Change db1051 IP - T156004 (duration: 00m 39s)

Mentioned in SAL (#wikimedia-operations) [2017-01-24T13:16:00Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Change db1051 IP - T156004 (duration: 00m 39s)

Marostegui closed this task as "Resolved".Jan 24 2017, 1:33 PM
Marostegui assigned this task to Cmjohnson.

db1051 has been moved.
DNS updated
db-eqiad,codfw files updated
mysql and replication started finely.
tendril updated

Thanks @Cmjohnson you are lighting fast!

Change 333907 had a related patch set uploaded (by Cmjohnson):
Updating dns for db1052 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333907

Change 333907 merged by Cmjohnson:
Updating dns for db1052 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333907

Change 333911 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Repool db1051 with less weight

https://gerrit.wikimedia.org/r/333911

Change 333911 merged by jenkins-bot:
db-eqiad.php: Repool db1051 with less weight

https://gerrit.wikimedia.org/r/333911

Mentioned in SAL (#wikimedia-operations) [2017-01-24T14:41:14Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1051 with less weight - T156004 (duration: 00m 41s)