Page MenuHomePhabricator

Migrate codfw row C & D database hosts to new Leaf switches
Closed, ResolvedPublic

Description

There are 58 database hosts connected to the private1-c-codfw and private1-d-codfw vlans on the legacy top-of-rack switches in codfw rows C and D. See the full list on P66916.

Similar to what we recently did for rows A and B (T355544) we need to move the network connection on all of these hosts from the current top-of-rack switch they are connected to (asw-xxxxx) to the newer equipment installed in parallel in each rack to replace them (lsw1-xxxx). One lesson from the experience in rows A and B is that approaching this on a rack-by-rack basis is tricky for the DBAs, as it is effectively a random order in terms of the database setup.

Each host can be moved individually, so for this move it is probably best we tackle the DB hosts as a unit, and come up with a schedule of master switchovers, depools and cable moves that overall is least disruptive.

DC-Ops have pre-cabled the switch side of the new links, so the downtime for each host will likely be only a few seconds (although we should probably be prepared for slightly longer). The links will also be moved one-by-one, so no two hosts will be down at any given time.

Details

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:18:16Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2206 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68655 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:18:18Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2031 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68656 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:18:20Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2036 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68657 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:42:22Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2125 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68658 and previous config saved to /var/cache/conftool/dbconfig/20240904-164221-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:42:44Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2138 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68659 and previous config saved to /var/cache/conftool/dbconfig/20240904-164243-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:43:06Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68660 and previous config saved to /var/cache/conftool/dbconfig/20240904-164305-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:43:22Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2190 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68661 and previous config saved to /var/cache/conftool/dbconfig/20240904-164321-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:43:41Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2206 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68662 and previous config saved to /var/cache/conftool/dbconfig/20240904-164340-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:43:52Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68663 and previous config saved to /var/cache/conftool/dbconfig/20240904-164351-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:44:05Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2031 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68664 and previous config saved to /var/cache/conftool/dbconfig/20240904-164404-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:44:21Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2032 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68665 and previous config saved to /var/cache/conftool/dbconfig/20240904-164421-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:44:36Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2036 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68666 and previous config saved to /var/cache/conftool/dbconfig/20240904-164435-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:57:27Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2125 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68667 and previous config saved to /var/cache/conftool/dbconfig/20240904-165727-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:57:49Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2138 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68668 and previous config saved to /var/cache/conftool/dbconfig/20240904-165749-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:58:12Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68669 and previous config saved to /var/cache/conftool/dbconfig/20240904-165811-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:58:28Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2190 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68670 and previous config saved to /var/cache/conftool/dbconfig/20240904-165827-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:58:46Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2206 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68671 and previous config saved to /var/cache/conftool/dbconfig/20240904-165846-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:58:57Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68672 and previous config saved to /var/cache/conftool/dbconfig/20240904-165857-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:59:10Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2031 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68673 and previous config saved to /var/cache/conftool/dbconfig/20240904-165909-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:59:27Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2032 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68674 and previous config saved to /var/cache/conftool/dbconfig/20240904-165926-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T16:59:42Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2036 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68675 and previous config saved to /var/cache/conftool/dbconfig/20240904-165941-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:12:33Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68676 and previous config saved to /var/cache/conftool/dbconfig/20240904-171232-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:12:55Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2138 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68677 and previous config saved to /var/cache/conftool/dbconfig/20240904-171254-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:13:17Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68678 and previous config saved to /var/cache/conftool/dbconfig/20240904-171317-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:13:33Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2190 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68679 and previous config saved to /var/cache/conftool/dbconfig/20240904-171332-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:13:52Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2206 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68680 and previous config saved to /var/cache/conftool/dbconfig/20240904-171351-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:14:03Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68681 and previous config saved to /var/cache/conftool/dbconfig/20240904-171402-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:14:15Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2031 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68682 and previous config saved to /var/cache/conftool/dbconfig/20240904-171415-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:14:32Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2032 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68683 and previous config saved to /var/cache/conftool/dbconfig/20240904-171431-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-04T17:14:47Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'es2036 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68684 and previous config saved to /var/cache/conftool/dbconfig/20240904-171447-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T16:20:58Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'depool db2141 db2144 db2150 db2169 db2186 db2191 - T370852', diff saved to https://phabricator.wikimedia.org/P68714 and previous config saved to /var/cache/conftool/dbconfig/20240905-162057-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T16:21:21Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: network maintenance T370852

Mentioned in SAL (#wikimedia-operations) [2024-09-05T16:21:38Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: network maintenance T370852

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:08:01Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68715 and previous config saved to /var/cache/conftool/dbconfig/20240905-170800-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:08:05Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2191 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68716 and previous config saved to /var/cache/conftool/dbconfig/20240905-170800-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:08:10Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2169 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68717 and previous config saved to /var/cache/conftool/dbconfig/20240905-170801-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:23:05Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68718 and previous config saved to /var/cache/conftool/dbconfig/20240905-172305-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:23:08Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2169 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68719 and previous config saved to /var/cache/conftool/dbconfig/20240905-172306-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:38:11Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68720 and previous config saved to /var/cache/conftool/dbconfig/20240905-173810-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:38:14Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2191 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68721 and previous config saved to /var/cache/conftool/dbconfig/20240905-173810-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:38:16Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2169 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68722 and previous config saved to /var/cache/conftool/dbconfig/20240905-173811-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:53:16Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68723 and previous config saved to /var/cache/conftool/dbconfig/20240905-175315-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:53:19Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2191 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68724 and previous config saved to /var/cache/conftool/dbconfig/20240905-175316-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-05T17:53:21Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2169 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68725 and previous config saved to /var/cache/conftool/dbconfig/20240905-175316-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-10T15:45:40Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'depool db2126 db2165 db2166 db2192 db2208 es2037 - T370852', diff saved to https://phabricator.wikimedia.org/P68813 and previous config saved to /var/cache/conftool/dbconfig/20240910-154540-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-11T15:41:14Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'depool db2115 db2116 db2127 db2167 db2168 db2179 db2180 db2210 es2022 es2038 - T370852', diff saved to https://phabricator.wikimedia.org/P68929 and previous config saved to /var/cache/conftool/dbconfig/20240911-154114-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T15:37:21Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'depool db2128 db2151 db2170 db2171 db2211 db2212 es2033 es2034 es2039 pc2014 db2209 - T370852', diff saved to https://phabricator.wikimedia.org/P69062 and previous config saved to /var/cache/conftool/dbconfig/20240912-153720-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-09-12T15:40:09Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'depool es2034 which was perceived master for es3 - T370852', diff saved to https://phabricator.wikimedia.org/P69063 and previous config saved to /var/cache/conftool/dbconfig/20240912-154008-arnaudb.json

Papaul claimed this task.

This is done