Now that T364095: Codfw row C/D switch installation & configuration is effectively completed we can begin the process of moving existing hosts from the old ASW switches in codfw rows C & D to the new Leaf switches there.
Schedule
The full schedule of planned maintenance windows is as listed below. On most days we will try to move the servers in two full racks, which seems a reasonable compromise between not disrupting too many hosts on a given day, and completing the moves in good time. It should allow us to get the work done before the datacentre switchover to codfw on September 25th.
| Racks | Date | Task |
|---|---|---|
| C1 | Wed Sept 4th 16:00 UTC | T373095 |
| C2 & C3 | Thu Sept 5th 16:30 UTC | T373096 |
| C4 & C5 | Tue Sep 10th 16:00 UTC | T373097 |
| C6 & C7 | Wed Sep 11th 16:00 UTC | T373101 |
| D1 & D2 | Thu Sep 12th 16:00 UTC | T373102 |
| D3 & D4 | Tue Sep 17th 16:00 UTC | T373103 |
| D5 & D6 | Wed Sep 18th 16:00 UTC | T373104 |
| D7 & D8 | Thu Sep 19th 16:00 UTC | T373105 |
Server links will be moved one-by-one from old to the new switch. So no two hosts will be offline at once.
Based on previous experience each host is likely to only lose comms for ~10 seconds. It is inevitable that a small number of the new cables do not work, however, or there is some minor glitch in the move. So it is possible in an edge case that a host will be offline for 2-3 minutes. On previous occasions this happened with about 1 out of 20 hosts.
Planning
We are using the below gsheet to plan the moves and detail actions that need to be taken to ensure things go smoothly:
https://docs.google.com/spreadsheets/d/16xoZuDeC_-o6s70uEMnvdgn4BlT1f8__WPYprRuduIA
SREs can help by:
- Listing the actions to be taken for a given type of host - if they can all be treated the same - on the first tab
- Listing the actions to be taken for a particular host - if we need to detail the plans for specific hosts - on the tab for each day
The Schedule by Team & Host Type tab lists all hosts, grouped by team and host type, with hosts of the same type to be moved on the same day highlighted.
The Schedule by Team & Date tab lists all the hosts, grouped by team and date, with all hosts to be moved on a given date highlighted.
Hopefully this will assist in planning for the moves.
Exceptions
We are planning the moves on a rack-by-rack schedule as this is the easiest way to group them and perform the actual work. However hosts can theoretically be moved in any order. So if there is a particular blocker for a given maintenance, for instance if two hosts are scheduled for the same day but they cannot both be depooled at the same time, we can try to move one or other ahead of the overall rack move, so they aren't both affected on the same day.
Any such exceptions can be noted here or please discuss with topranks on irc.