Page MenuHomePhabricator
Create Task
Maniphest T355861

Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw
Closed, ResolvedPublic
Actions

Description

Currently scheduled for Wed Feb 7 16:00 UTC

The following server uplink moves need to be completed as part of the wider migration from our old top-of-rack switches in codfw to their new replacements. The work is just to move the cable, so we expect an interruption of 60 seconds or less per hosts. Moves will be sequential, so only 1 host will be disconnected at any given moment.

TeamHost typeasw-a2-codfw intlsw1-a2-codfw int
Data Persistencems-be2044xe-2/0/4ms-be2044
Data Persistencems-be2074xe-2/0/9ms-be2074
Data Persistencems-be2051xe-2/0/11ms-be2051
Data Persistencems-fe2009xe-2/0/14ms-fe2009
Data Persistencems-fe2013xe-2/0/31ms-fe2013
Data Persistencethanos-fe2001xe-2/0/3thanos-fe2001
Infra Foundationsganeti2029xe-2/0/27hosts
Infra Foundationsganeti2030xe-2/0/28hosts
Machine Learningml-cache2001xe-2/0/8ml-cache2001
Observabilitykafka-logging2001xe-2/0/5kafka-logging2001
Observabilitylogging-hd2001xe-2/0/32logging-hd2001
Search Platformelastic2087xe-2/0/0elastic2087
Search Platformelastic2037xe-2/0/1elastic2037
Search Platformelastic2038xe-2/0/2elastic2038
Search Platformelastic2055xe-2/0/12elastic2055
Search Platformelastic2088xe-2/0/13elastic2088
Search Platformelastic2073xe-2/0/29elastic2073
Search Platformelastic2074xe-2/0/30elastic2074
Search Platformwdqs2013xe-2/0/10wdqs2013
Search Platformwdqs2023xe-2/0/19wdqs2023
Service Opsmc2039xe-2/0/7mc2039
Service Opsmc2038xe-2/0/26mc2038
moss-be2001xe-2/0/6moss-be2001

We can track the details of the moves and what needs to be done to prepare in the Google sheet here, if not specific action is needed for a given type of host just state that on the first tab

https://docs.google.com/spreadsheets/d/1PlGGLclKFYR9XaqjOLibhiwwny0fOD8gLMwsNhIzGRo

Related Objects
Search...

Event Timeline

cmooney triaged this task as Medium priority.Jan 25 2024, 11:36 AM
cmooney created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2024, 11:36 AM
cmooney added a parent task: T355617: Migrate cloudelastic from public to private IPs.Jan 25 2024, 11:38 AM
cmooney edited parent tasks, added: T355544: Migrate hosts from codfw row A/B ASW to new LSW devices; removed: T355617: Migrate cloudelastic from public to private IPs.Jan 25 2024, 11:40 AM
cmooney mentioned this in T355544: Migrate hosts from codfw row A/B ASW to new LSW devices.Jan 25 2024, 12:22 PM
Jhancock.wm moved this task from Backlog to Codfw Switch migration on the ops-codfw board.Jan 25 2024, 4:58 PM
klausman subscribed.Jan 30 2024, 9:46 AM
MatthewVernon added a project: SRE-swift-storage.Jan 30 2024, 11:16 AM
MatthewVernon subscribed.

swift will need depooling in codfw before this work.
Likewise the affected thanos-fe node.

I'm oncall that week, so if there's an outage at the time I may have to fling a spanner in the works. Hopefully not necessary!

akosiaris subscribed.Feb 5 2024, 8:14 PM
Jhancock.wm added a comment.Feb 6 2024, 2:07 PM

This rack is physically ready for tomorrow.

klausman added a subtask: T356873: Downtime ml-cache2001 for network link move.Feb 7 2024, 2:43 PM
Stashbot added a comment.Feb 7 2024, 2:44 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T14:44:25Z] <klausman@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on ml-cache2001.codfw.wmnet with reason: Machine network link move (T355861)

ops-monitoring-bot mentioned this in T356873: Downtime ml-cache2001 for network link move.Feb 7 2024, 2:44 PM
Stashbot added a comment.Feb 7 2024, 2:44 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T14:44:42Z] <klausman@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ml-cache2001.codfw.wmnet with reason: Machine network link move (T355861)

klausman closed subtask T356873: Downtime ml-cache2001 for network link move as Resolved.Feb 7 2024, 2:45 PM
ops-monitoring-bot added a comment.Feb 7 2024, 3:46 PM

Icinga downtime and Alertmanager silence (ID=bab7a949-b7c4-40b3-b9f5-e00978a8ce0f) set by cmooney@cumin1002 for 1:00:00 on 4 host(s) and their services with reason: prepping for server uplink migration codfw rack a2

asw-a-codfw,cr[1-2]-codfw,lsw1-a2-codfw.mgmt
MatthewVernon added a comment.Feb 7 2024, 3:47 PM

You're good to go re swift and thanos now.

cmooney added a comment.Feb 7 2024, 3:48 PM
In T355861#9521522, @MatthewVernon wrote:

You're good to go re swift and thanos now.

Thanks!

ops-monitoring-bot added a comment.Feb 7 2024, 3:59 PM

Icinga downtime and Alertmanager silence (ID=075fbfb6-a879-438b-b065-55d67628e920) set by cmooney@cumin1002 for 0:30:00 on 22 host(s) and their services with reason: Migrating servers in codfw rack A2 to lsw1-a2-codfw

elastic[2037-2038,2055,2073-2074,2087-2088].codfw.wmnet,ganeti[2029-2030].codfw.wmnet,kafka-logging2001.codfw.wmnet,logging-hd2001.codfw.wmnet,lvs2011.codfw.wmnet,mc[2038-2039].codfw.wmnet,ml-cache2001.codfw.wmnet,ms-be[2051,2074].codfw.wmnet,ms-fe[2009,2013].codfw.wmnet,thanos-fe2001.codfw.wmnet,wdqs[2013,2023].codfw.wmnet
Stashbot added a comment.Feb 7 2024, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T16:04:51Z] <vgutierrez> <topranks> Commencing server uplink moves from old switch to new in codfw rack A2 T355861

Stashbot added a comment.Feb 7 2024, 4:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T16:16:02Z] <Emperor> repool thanos-fe2001 T355861

Stashbot added a comment.Feb 7 2024, 4:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T16:16:27Z] <Emperor> repool codfw dnsdisc T355861

cmooney added a comment.Feb 7 2024, 4:39 PM

All servers moved and pinging again fine. Very short delay for all apart from elastic2038 (issue there due to badly seated NIC card shifting when cable was moved, needed to be properly re-inserted into motherboard after which all ok).

Any systems that were depooled/drained can now be brought back live. Many thanks to everyone for their help!

MoritzMuehlenhoff subscribed.Feb 8 2024, 8:03 AM

I've kicked off a rebalance of ganeti/A now that the maintenance is over.

cmooney closed this task as Resolved.Feb 8 2024, 9:43 AM
cmooney claimed this task.
In T355861#9523826, @MoritzMuehlenhoff wrote:

I've kicked off a rebalance of ganeti/A now that the maintenance is over.

Thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL