Those servers currently co-exist in the same racks with their stand-by servers, which is not great.
Please re-rack the following machines:
#1 mc-gp2006 : from D4 → D3
#2 mc2046: from B7 → B8
A heads up in -sre-private would be nice. You may do this work any time as long as those those two servers are not offline simultaneously.
I can re-rack mc-gp2006 in U-26 of D3 on Wednesday.
rack B5 is not a good location to rack mc2046 because of 10G port availability on that switch. B8 or B3 would work better.
I can do that on Thursday just to be extra sure down time doesn't overlap.
@jijiki i got the server moved. the mgmt pings.
lmk if you have a preference for mc2046
FYI, there was this outstanding diff:
Change for lsw1-d4-codfw.mgmt.codfw.wmnet: [edit interfaces xe-0/0/43] - description mc-gp2006;
It's because the cable got removed, but the old interface was still enabled.
I disabled the interface (https://netbox.wikimedia.org/extras/changelog/288031/) then ran Homer on the switch.
Change #1314701 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] mcrouter_wancache: temporary remove mc-gp2006
Thank you Jenn! We need to re-IP mc-gp2006 as it turns out, which means that moving mc2046 can happen after that. Do you have time this week help me with this?
B8 for mc2046 will be great, I have updated the description
Change #1314701 merged by Effie Mouzeli:
[operations/puppet@production] mcrouter_wancache: temporary remove mc-gp2006
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm completed:
@Jhancock.wm mc-gp2006 is alive an well (tx @cmooney for sorting out the networking bits). Is it alright if we re-rack mc2046 sometime next week?
Change #1315946 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] mcrouter_wancache: re-add mc-gp2006 to the gutter-pool
Change #1315946 merged by Effie Mouzeli:
[operations/puppet@production] mcrouter_wancache: re-add mc-gp2006 to the gutter-pool
@jijiki it's moved. but unless you want to reimage it, you'll need to redo whatever magic cathal did.
Networking is no magic :)
The server is still connected to the B7 rack though, what is its new switch port ?
For the re-image, the "hack" is to set the host's switch port vlan to one of the LEGACY_VLANS https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/hosts/__init__.py#L38 (like it's currently configured, but on the wrong old B7 switch port)
Then run the re-image cookbook with --move-vlan
it's magical to me. Updated the server location. I was gonna give the reimage a try but i didn't know which distro was on it.
@jijiki lmk which distro or you can take over.
I made the change now. @Jhancock.wm if you need to do this again the trick Arzhel is suggesting is when you connect the new switch port to the server in Netbox, set the "untagged vlan" on the new port to one of the row-wide ones. In this case I used "private1-b-codfw".
When the reimage cookbook is run with --move-vlan, and it sees the port is on one of those, it will change the port vlan to a rack-specific one, say private1-b8-codfw. But it will also assign new IPs for the host and fix up the dns (the bit I did manually last time). So it can save a few steps when we do server moves like this.
No hassle we are always around if needed, but that's the quickest way to do it.
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc2046.codfw.wmnet with OS trixie
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc2046.codfw.wmnet with OS trixie completed:
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc2046.codfw.wmnet with OS trixie
Change #1321564 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] mcrouter_wancache: replace mc2046's IP
Change #1321564 merged by Effie Mouzeli:
[operations/puppet@production] mcrouter_wancache: replace mc2046's IP
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc2046.codfw.wmnet with OS trixie completed:
Thank you @Jhancock.wm and @cmooney for doing all the magic stuff, server is serving :)