Page MenuHomePhabricator

Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw
Closed, ResolvedPublic

Description

Currently scheduled for Thur Feb 8 16:00 UTC

The following server uplink moves need to be completed as part of the wider migration from our old top-of-rack switches in codfw to their new replacements. The work is just to move the cable, so we expect an interruption of 60 seconds or less per hosts. Moves will be sequential, so only 1 host will be disconnected at any given moment.

TeamHost typeasw-a3-codfw intlsw1-a3-codfw int
Data persistencedb2142ge-3/0/16move
Data persistencedb2103ge-3/0/23move
Data Persistencees2020ge-3/0/27es2020
Service Opsmw2377ge-3/0/0depool
Service Opsmw2378ge-3/0/1depool
Service Opsmw2379ge-3/0/2depool
Service Opsmw2380ge-3/0/3depool
Service Opsmw2381ge-3/0/4depool
Service Opsmw2382ge-3/0/5depool
Service Opsmw2383ge-3/0/6depool
Service Opsmw2384ge-3/0/7depool
Service Opsmw2385ge-3/0/8depool
Service Opsmw2386ge-3/0/9depool
Service Opsmw2387ge-3/0/10depool
Service Opsmw2388ge-3/0/11depool
Service Opsmw2389ge-3/0/12depool
Service Opsmw2390ge-3/0/13depool
Service Opsmw2391ge-3/0/14depool
Service Opsmw2392ge-3/0/15depool
Service Opsmw2393ge-3/0/17depool
Service Opsmw2394ge-3/0/18depool
Service Opsmw2395ge-3/0/19depool
Service Opsmw2396ge-3/0/20depool
Service Opsmw2397ge-3/0/21depool
Service Opsmw2398ge-3/0/22depool
Service Opsmw2399ge-3/0/24depool
Service Opsmw2400ge-3/0/25depool
Service Opsmw2291ge-3/0/28depool
Service Opsmw2292ge-3/0/29depool
Service Opsmw2293ge-3/0/30depool
Service Opsmw2294ge-3/0/31depool
Service Opsmw2295ge-3/0/32depool
Service Opsmw2296ge-3/0/33depool
Service Opsmw2297ge-3/0/34depool
Service Opsmw2298ge-3/0/35depool
Service Opsmw2299ge-3/0/36depool
Service Opsmw2300ge-3/0/37depool
Infra Foundations / Observabilitynetmon2002ge-3/0/26ok

We can track the details of the moves and what needs to be done to prepare in the Google sheet here, if not specific action is needed for a given type of host just state that on the first tab

https://docs.google.com/spreadsheets/d/1PlGGLclKFYR9XaqjOLibhiwwny0fOD8gLMwsNhIzGRo

Event Timeline

cmooney triaged this task as Medium priority.Jan 25 2024, 11:37 AM
cmooney created this task.

db2142 - x2 master
db2103 - s1 master
es2020 - es4 master

db2142 - x2 master
db2103 - s1 master
es2020 - es4 master

These are no longer masters and can be done anytime. db2103, es2020 need depooling. db2142 is a slave but not used.

Mentioned in SAL (#wikimedia-operations) [2024-02-08T05:53:17Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2103 es2020 T355862', diff saved to https://phabricator.wikimedia.org/P56487 and previous config saved to /var/cache/conftool/dbconfig/20240208-055316-root.json

The databases are ready to be moved any time.

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:49:45Z] <claime> Draining mw2377.codfw.wmnet mw2378.codfw.wmnet mw2381.codfw.wmnet mw2395.codfw.wmnet mw2291.codfw.wmnet mw2292.codfw.wmnet mw2293.codfw.wmnet mw2294.codfw.wmnet mw2295.codfw.wmnet mw2296.codfw.wmnet mw2297.codfw.wmnet - T355862

The databases are ready to be moved any time.

Great, thanks!

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:56:08Z] <claime> Depooled mw2379|mw2380|mw2382|mw2383|mw2384|mw2385|mw2386|mw2387|mw2388|mw2389|mw2390|mw2391|mw2392|mw2393|mw2394|mw2396|mw2397|mw2398|mw2399|mw2400|mw2298|mw2299|mw2300 - T355862

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:57:37Z] <topranks> moving Netbox server uplinks from asw-a3-codfw to lsw1-a3-codfw to prep config for server moves T355862

Mentioned in SAL (#wikimedia-operations) [2024-02-08T16:07:44Z] <topranks> Commencing server uplink moves from old switch to new in codfw rack A3 T355862

Icinga downtime and Alertmanager silence (ID=a24ae7f4-1952-434f-9ee8-3ff0973f1444) set by cmooney@cumin1002 for 1:00:00 on 4 host(s) and their services with reason: server uplink migration codfw rack a3

asw-a-codfw,cr[1-2]-codfw,lsw1-a3-codfw.mgmt

Icinga downtime and Alertmanager silence (ID=06c4fbb3-382e-4660-b308-79bf9f5106d5) set by cmooney@cumin1002 for 0:30:00 on 39 host(s) and their services with reason: Migrating servers in codfw rack A3 to lsw1-a3-codfw

db[2103,2142,2158].codfw.wmnet,es2020.codfw.wmnet,mw[2291-2300,2377-2400].codfw.wmnet,netmon2002.wikimedia.org

Mentioned in SAL (#wikimedia-operations) [2024-02-08T16:23:11Z] <topranks> Server move completed codfw rack A3 T355862

Work completed! No errors to report all working well.

Thanks - I am starting to repool the databases.

Mentioned in SAL (#wikimedia-operations) [2024-02-08T16:40:42Z] <claime> Uncordoning mw2377.codfw.wmnet mw2378.codfw.wmnet mw2381.codfw.wmnet mw2395.codfw.wmnet mw2291.codfw.wmnet mw2292.codfw.wmnet mw2293.codfw.wmnet mw2294.codfw.wmnet mw2295.codfw.wmnet mw2296.codfw.wmnet mw2297.codfw.wmnet - T355862

Mentioned in SAL (#wikimedia-operations) [2024-02-08T16:48:04Z] <claime> Repooling mw2379|mw2380|mw2382|mw2383|mw2384|mw2385|mw2386|mw2387|mw2388|mw2389|mw2390|mw2391|mw2392|mw2393|mw2394|mw2396|mw2397|mw2398|mw2399|mw2400|mw2298|mw2299|mw2300 - T355862

cmooney claimed this task.

Thanks everyone for the help on getting this done!