Page MenuHomePhabricator

Migrate servers in codfw racks C6 & C7 from asw to lsw
Closed, ResolvedPublic

Description

Currently scheduled for Wed Sept 11th 2024 16:00 UTC

As part of the scheduled refresh of switch equipment in codfw rows C and D we need to move the network connections for servers in racks C6 and C7 from the old to new switch.

Hosts in this rack are managed by the following teams:

Data Persistence
Infrastructure Foundations
Observability
Search Platform
ServiceOps
Traffic

A full list of the specific hosts can be found below. We will use the sheet to plan the moves and co-ordinate with other SRE teams on actions required to ensure things go smoothly:

https://docs.google.com/spreadsheets/d/16xoZuDeC_-o6s70uEMnvdgn4BlT1f8__WPYprRuduIA#gid=1162497652

Server links will be moved one-by-one from old to the new switch. So no two hosts will be offline at once.

Based on previous experience each host is likely to only lose comms for ~10 seconds. It is inevitable that a small number of the new cables do not work, however, or there is some minor glitch in the move. So it is possible in an edge case that a host will be offline for 2-3 minutes. On previous occasions this happened with about 1 out of 20 hosts.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:09Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2167 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68966 and previous config saved to /var/cache/conftool/dbconfig/20240911-171709-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:14Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68967 and previous config saved to /var/cache/conftool/dbconfig/20240911-171714-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:19Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2179 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68968 and previous config saved to /var/cache/conftool/dbconfig/20240911-171719-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:25Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2180 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68969 and previous config saved to /var/cache/conftool/dbconfig/20240911-171724-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:30Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2210 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68970 and previous config saved to /var/cache/conftool/dbconfig/20240911-171729-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:34Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2022 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68971 and previous config saved to /var/cache/conftool/dbconfig/20240911-171734-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:39Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2038 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68972 and previous config saved to /var/cache/conftool/dbconfig/20240911-171739-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T15:37:41Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 0:45:00 on 11 hosts with reason: network maintenance T373101

Mentioned in SAL (#wikimedia-operations) [2024-09-12T15:38:00Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:45:00 on 11 hosts with reason: network maintenance T373101

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:20Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69064 and previous config saved to /var/cache/conftool/dbconfig/20240912-161916-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:24Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2151 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69065 and previous config saved to /var/cache/conftool/dbconfig/20240912-161922-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:28Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2170 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69066 and previous config saved to /var/cache/conftool/dbconfig/20240912-161927-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:33Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2171 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69067 and previous config saved to /var/cache/conftool/dbconfig/20240912-161932-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:38Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2211 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69068 and previous config saved to /var/cache/conftool/dbconfig/20240912-161937-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:43Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69069 and previous config saved to /var/cache/conftool/dbconfig/20240912-161942-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:48Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2033 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69070 and previous config saved to /var/cache/conftool/dbconfig/20240912-161947-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:53Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2034 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69071 and previous config saved to /var/cache/conftool/dbconfig/20240912-161952-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:19:58Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2039 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69072 and previous config saved to /var/cache/conftool/dbconfig/20240912-161957-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:20:08Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2209 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69073 and previous config saved to /var/cache/conftool/dbconfig/20240912-162007-arnaudb.json

cmooney claimed this task.

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:22Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69075 and previous config saved to /var/cache/conftool/dbconfig/20240912-163422-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:28Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2151 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69076 and previous config saved to /var/cache/conftool/dbconfig/20240912-163427-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:33Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2170 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69077 and previous config saved to /var/cache/conftool/dbconfig/20240912-163433-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:38Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2171 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69078 and previous config saved to /var/cache/conftool/dbconfig/20240912-163438-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:43Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2211 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69079 and previous config saved to /var/cache/conftool/dbconfig/20240912-163443-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:49Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69080 and previous config saved to /var/cache/conftool/dbconfig/20240912-163448-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:54Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2033 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69081 and previous config saved to /var/cache/conftool/dbconfig/20240912-163453-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:34:58Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2034 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69082 and previous config saved to /var/cache/conftool/dbconfig/20240912-163458-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:35:04Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2039 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69083 and previous config saved to /var/cache/conftool/dbconfig/20240912-163503-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:35:13Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2209 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69084 and previous config saved to /var/cache/conftool/dbconfig/20240912-163513-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:49:28Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69085 and previous config saved to /var/cache/conftool/dbconfig/20240912-164927-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:49:34Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2151 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69086 and previous config saved to /var/cache/conftool/dbconfig/20240912-164933-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:49:39Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2170 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69087 and previous config saved to /var/cache/conftool/dbconfig/20240912-164938-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:49:44Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2171 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69088 and previous config saved to /var/cache/conftool/dbconfig/20240912-164943-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:49:48Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2211 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69089 and previous config saved to /var/cache/conftool/dbconfig/20240912-164948-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:49:54Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2212 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69090 and previous config saved to /var/cache/conftool/dbconfig/20240912-164953-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:49:59Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2033 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69091 and previous config saved to /var/cache/conftool/dbconfig/20240912-164959-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:50:04Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2034 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69092 and previous config saved to /var/cache/conftool/dbconfig/20240912-165003-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:50:09Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2039 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69093 and previous config saved to /var/cache/conftool/dbconfig/20240912-165009-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T16:50:19Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2209 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69094 and previous config saved to /var/cache/conftool/dbconfig/20240912-165018-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:04:34Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69096 and previous config saved to /var/cache/conftool/dbconfig/20240912-170433-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:04:39Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2151 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69097 and previous config saved to /var/cache/conftool/dbconfig/20240912-170439-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:04:45Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2170 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69098 and previous config saved to /var/cache/conftool/dbconfig/20240912-170444-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:04:51Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2171 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69099 and previous config saved to /var/cache/conftool/dbconfig/20240912-170449-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:04:54Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2211 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69100 and previous config saved to /var/cache/conftool/dbconfig/20240912-170453-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:05:00Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2212 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69101 and previous config saved to /var/cache/conftool/dbconfig/20240912-170459-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:05:05Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2033 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69102 and previous config saved to /var/cache/conftool/dbconfig/20240912-170504-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:05:09Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2034 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69103 and previous config saved to /var/cache/conftool/dbconfig/20240912-170509-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:05:15Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2039 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69104 and previous config saved to /var/cache/conftool/dbconfig/20240912-170514-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:05:25Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'pc2014 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69105 and previous config saved to /var/cache/conftool/dbconfig/20240912-170524-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:05:28Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2209 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69106 and previous config saved to /var/cache/conftool/dbconfig/20240912-170524-arnaudb.json