Page MenuHomePhabricator

Physically move db1131 from B5 to C8
Closed, ResolvedPublic

Description

In order to spread our masters better within rows and racks, I would like to know if we can physically move db1131 from B5 to C8.
The idea would be to move it and then we can failover db1093 (s6 primary master) to it, so we can decommission it (in Q2 will need to be decommissioned).

@wiki_willy @Cmjohnson how doable is this within the next 2 weeks? So we can failover the primary master while eqiad is passive so we don't have to have downtime?.

What would be needed from DC-OPs would be:

  • Move the server physically
  • Provide a new IP within row C

Can you guys let me know if this is something you can squeeze within your priorities for the next two weeks?
Thank you!

Event Timeline

Marostegui renamed this task from Physically db1131 from B5 to C8 to Physically move db1131 from B5 to C8.Sep 15 2020, 8:55 AM

@Cmjohnson - are you able to work with @Marostegui on moving this host, maybe after one of the PDU upgrades this week? Thanks, Willy

@Marostegui is going to schedule this down tomorrow 17 Sept to be relocated to C8

@Cmjohnson mysql is down on this host. I haven't powered it off yet, just to see if you can give me the new IP before I do so, so I can change it on the host so it boots up directly with the new gateway and ip address.

Change 628128 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] updating dns for db1131

https://gerrit.wikimedia.org/r/628128

Change 628128 merged by Cmjohnson:
[operations/dns@master] updating dns for db1131

https://gerrit.wikimedia.org/r/628128

The server has been moved, dns entries were updated, updated asw2-c and removed from asw2-b. I am able to ping both mgmt and production. resolving

Mentioned in SAL (#wikimedia-operations) [2020-09-17T15:59:55Z] <marostegui> Update rack location on zarcillo for db1131 T262901

Mentioned in SAL (#wikimedia-operations) [2020-09-17T16:03:14Z] <marostegui> Recreate db1131 on tendril T262901

Mentioned in SAL (#wikimedia-operations) [2020-09-17T16:05:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Change db1131 IP after moving it to a different rack T262901', diff saved to https://phabricator.wikimedia.org/P12639 and previous config saved to /var/cache/conftool/dbconfig/20200917-160540-marostegui.json

Thank you so much Chris.
I am waiting for the host to catch up before repooling.
Tendril and zarcillo have been updated.

Change 628250 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1131: Enable notifications

https://gerrit.wikimedia.org/r/628250

Change 628250 merged by Marostegui:
[operations/puppet@production] db1131: Enable notifications

https://gerrit.wikimedia.org/r/628250