Page MenuHomePhabricator
Create Task
Maniphest T156004

Move db1051 to row B3
Closed, ResolvedPublic
Actions

Description

Hey @Cmjohnson

We would need db10151 to be moved to B3 if possible. This is in order to start mitigating the problem with the switch that got rebooted twice in the last two weeks (T155999).

If possible, can you also update DNS for us?
If you happen to know the IP it will get beforehand, we can change it before we poweroff the server so it will boot up with the correct one.
No worries if you don't, we will go thru ILO and get it changed.

Coordinate with me before the move as we need to depool the server.

Thanks!
Manuel.

Details

SubjectRepoBranchLines +/-
db-eqiad.php: Repool db1051 with less weightoperations/mediawiki-configmaster+12 -12
db-codfw,db-eqiad.php: Update db1051 IPoperations/mediawiki-configmaster+2 -2
db-eqiad.php: Depool db1051operations/mediawiki-configmaster+4 -4
db-eqiad.php: Depool db1051operations/mediawiki-configmaster+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 23 2017, 3:15 PM
Marostegui updated the task description. (Show Details)Jan 23 2017, 3:20 PM
Marostegui mentioned this in T156006: Move db1052 to row B3.
Marostegui mentioned this in T156008: Switchover s1 master db1057 -> db1052.Jan 23 2017, 3:24 PM
Marostegui mentioned this in T155999: DBA plan to mitigate asw-c2-eqiad reboots.
faidon added a comment.Jan 23 2017, 3:41 PM

D 2 is a potentially bad choice, as it will soon become a 10G rack once the T148506 migration is (finally…) done. Any other rack except D 2 & D 7 (new 10G) and D 6-8 (current 10G) is probably a safe bet.

Marostegui added a comment.Jan 23 2017, 3:46 PM

Thanks Faidon - let's go for B3 then?

Marostegui renamed this task from Move db1051 to row D2 to Move db1051 to row B3.Jan 23 2017, 3:48 PM
Marostegui updated the task description. (Show Details)
faidon added a comment.Jan 23 2017, 3:48 PM

Sure, no objections from a networking/DC design perspective :)

Marostegui added a comment.Jan 24 2017, 9:25 AM

Alerts silenced for 24 hours - I will re-enable them once the move is done.

gerritbot subscribed.Jan 24 2017, 10:17 AM

Change 333873 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333873

gerritbot added a project: Patch-For-Review.Jan 24 2017, 10:17 AM
gerritbot added a comment.Jan 24 2017, 12:47 PM

Change 333873 merged by Marostegui:
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333873

Stashbot subscribed.Jan 24 2017, 12:48 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-24T12:48:33Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1051 - T156004 (duration: 00m 39s)

gerritbot added a comment.Jan 24 2017, 12:49 PM

Change 333898 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333898

gerritbot added a comment.Jan 24 2017, 12:53 PM

Change 333898 merged by jenkins-bot:
db-eqiad.php: Depool db1051

https://gerrit.wikimedia.org/r/333898

Stashbot added a comment.Jan 24 2017, 12:55 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-24T12:55:01Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1051 - T156004 (duration: 00m 39s)

Stashbot added a comment.Jan 24 2017, 12:56 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-24T12:56:29Z] <marostegui> Shutdown mysql on db1051 for maintenance - T156004

gerritbot added a comment.Jan 24 2017, 1:00 PM

Change 333899 had a related patch set uploaded (by Cmjohnson):
Updating dns for db1051 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333899

Stashbot added a comment.Jan 24 2017, 1:00 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-24T13:00:44Z] <marostegui> Shutdown db1051 for maintenance - T156004

gerritbot added a comment.Jan 24 2017, 1:01 PM

Change 333899 merged by Cmjohnson:
Updating dns for db1051 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333899

gerritbot added a comment.Jan 24 2017, 1:05 PM

Change 333900 had a related patch set uploaded (by Marostegui):
db-codfw,db-eqiad.php: Update db1051 IP

https://gerrit.wikimedia.org/r/333900

gerritbot added a comment.Jan 24 2017, 1:11 PM

Change 333900 merged by jenkins-bot:
db-codfw,db-eqiad.php: Update db1051 IP

https://gerrit.wikimedia.org/r/333900

Stashbot added a comment.Jan 24 2017, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-24T13:14:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Change db1051 IP - T156004 (duration: 00m 39s)

Stashbot added a comment.Jan 24 2017, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-24T13:16:00Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Change db1051 IP - T156004 (duration: 00m 39s)

Marostegui closed this task as Resolved.Jan 24 2017, 1:33 PM
Marostegui assigned this task to • Cmjohnson.

db1051 has been moved.
DNS updated
db-eqiad,codfw files updated
mysql and replication started finely.
tendril updated

Thanks @Cmjohnson you are lighting fast!

gerritbot added a comment.Jan 24 2017, 1:42 PM

Change 333907 had a related patch set uploaded (by Cmjohnson):
Updating dns for db1052 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333907

gerritbot added a comment.Jan 24 2017, 1:42 PM

Change 333907 merged by Cmjohnson:
Updating dns for db1052 to coincide with rack change T156004

https://gerrit.wikimedia.org/r/333907

gerritbot added a comment.Jan 24 2017, 2:21 PM

Change 333911 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Repool db1051 with less weight

https://gerrit.wikimedia.org/r/333911

gerritbot added a comment.Jan 24 2017, 2:39 PM

Change 333911 merged by jenkins-bot:
db-eqiad.php: Repool db1051 with less weight

https://gerrit.wikimedia.org/r/333911

Stashbot added a comment.Jan 24 2017, 2:41 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-24T14:41:14Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1051 with less weight - T156004 (duration: 00m 41s)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits