Page MenuHomePhabricator

Move some masters away from B6
Closed, ResolvedPublic

Description

We just realised that we have all the codfw primary masters in the same rack, B6:

db2016 (can be decommissioned)
db2017
db2018
db2019
db2023 (can be decommissioned)
db2028
db2029

We would like to spread them across racks for obvious reasons :-)
This would be my proposal:

db2017 -> B8
db2028 -> A1
db2029 -> C5
db2023 -> D5 (can be decommissioned) - not needed
db2018 -> D8

@Papaul let me know if this doable
Thanks!

Details

Related Gerrit Patches:
operations/dns : masterDNS: Change production DNS for db2051
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Change db2051 IP

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptJul 3 2017, 9:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Jul 3 2017, 9:09 AM
Marostegui moved this task from Triage to Next on the DBA board.
ayounsi added a subscriber: ayounsi.Jul 3 2017, 9:10 AM
jcrespo added a subscriber: jcrespo.Jul 3 2017, 9:13 AM

We may want to hold this, at least unless a switch is planned- hosts <db2030 will be decomissioned soon, so the masters will be failed over soon, probably.

Marostegui changed the task status from Open to Stalled.Jul 3 2017, 9:16 AM
Marostegui moved this task from Next to Blocked external/Not db team on the DBA board.
Papaul added a comment.Jul 3 2017, 3:19 PM

@Marostegui Proposal approved.

Thanks @Papaul - let's leave this stalled for now. We will ping you if we decide to go for it :-)

Hi @Papaul

We are planning to switchover db2019, and we will promote db2051 to master which is in C6, but we'd like to move it to B8 for instance?
Could you let us know which day works for you to get it moved?

Thanks

Papaul added a comment.Aug 1 2017, 2:38 PM

Hi @Marostegui

We can do this tomorrow with no problem.

Hi @Marostegui
We can do this tomorrow with no problem.

Sounds good, any specific time?

Papaul added a comment.Aug 1 2017, 3:05 PM

10:00 am CDT

Excellent, I will get the server ready by then, just to change the IP on it and power it off :)
Thanks a lot!

ayounsi removed a subscriber: ayounsi.Aug 1 2017, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-02T15:06:16Z] <marostegui> Poweroff db2051 to get it move to another rack - T169501

Change 369673 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Change db2051 IP

https://gerrit.wikimedia.org/r/369673

Papaul added a comment.Aug 2 2017, 3:13 PM

New switch port configuration for db2051

asw-b8-codfw ge-8/0/17

Change 369673 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Change db2051 IP

https://gerrit.wikimedia.org/r/369673

Mentioned in SAL (#wikimedia-operations) [2017-08-02T15:17:33Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Change db2051 IP - T169501 (duration: 00m 46s)

Change 369675 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Change production DNS for db2051

https://gerrit.wikimedia.org/r/369675

Mentioned in SAL (#wikimedia-operations) [2017-08-02T15:18:21Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Change db2051 IP - T169501 (duration: 00m 46s)

Change 369675 merged by Marostegui:
[operations/dns@master] DNS: Change production DNS for db2051

https://gerrit.wikimedia.org/r/369675

Marostegui updated the task description. (Show Details)Nov 27 2017, 12:58 PM

db2023 and db2016 aren't masters anymore, so they can be decommissioned.
We will create a decommission task when ready

@Marostegui Assigning this back to you . Just assign me the decommissioned task when ready.

Thanks.

Papaul reassigned this task from Papaul to Marostegui.Jan 16 2018, 3:59 PM
Marostegui closed this task as Resolved.Jan 16 2018, 4:01 PM
Marostegui reassigned this task from Marostegui to jcrespo.

I believe with all the work @jcrespo has done switching over servers this is resolved

This will be handled on T184090, which is not ready yet for papaul (but will be soon).