Change Details

Now that {T364095} is effectively completed we can begin the process of moving existing hosts from the old ASW switches in codfw rows C & D to the new Leaf switches there. **Order of moves** We probably need to approach this in several phases to ensure it goes smoothly and minimise the disruption to other teams and live services during the work. One lesson from T355544, when we moved the servers from the old switches to new in rows A & B, is that for certain clusters of hosts it is a lot more difficult to schedule than others. In brief, for some clusters of hosts providing a particular service, they are all basically "equal", and depooling any set of N hosts requires the same effort and analysis as any other random set of N hosts. In those cases moving the hosts on a rack-by-rack basis works just fine, it doesn't make a difference to the service running on them what order things are depooled. Other services, however, are not like that. For instance if specific hosts in the cluster are performing certain duties (master nodes etc), then moving the hosts in an effectively random order (rack-by-rack) can be difficult to orchestrate for the service owners. The database hosts maintained by our Data Platform were particularly tricky to get moved the last time. The best way to proceed would therefore seem to be: # Move LVS hosts as they need to connect to the new per-rack vlans before we start # Move hosts for services that may require special attention on a host-by-host basis (i.e. db's) # Move remaining hosts on a rack-by-rack basis **Planning** We probably need to set up some meetings with service owners who maintain hosts that fit into category two. To begin with I think we need to speak to: * Traffic (to discuss LVS plan) * Data Platform (to discuss DBs) * Data Persistence (swift, kafka etc) * Service Ops (most of these hosts can be done rack-by-rack) * Search (again I think can be done rack-by-rack) * Others TBC Once we've identified any sets of hosts that need special treatment we can create sub-tasks to plan the move of those hosts. We probably also need to work on some automation to move specific hosts - rather than an entire rack - from one switch to another. Lastly we need to create the per-rack move tasks for the remaining hosts, set up a schedule for the moves, and work with all the SREs to get the work completed.

Now that {T364095} is effectively completed we can begin the process of moving existing hosts from the old ASW switches in codfw rows C & D to the new Leaf switches there. **Hosts** Hosts on the following vlans need to be moved (or see [[ https://phabricator.wikimedia.org/P66915 | full list ]]): |Vlan|Hosts| |-----|-------| |private1-c-codfw|P66913| |public1-c-codfw|P66911| |private1-d-codfw|P66914| |public1-d-codfw|P66912| **Order of moves** We probably need to approach this in several phases to ensure it goes smoothly and minimise the disruption to other teams and live services during the work. One lesson from T355544, when we moved the servers from the old switches to new in rows A & B, is that for certain clusters of hosts it is a lot more difficult to schedule than others. In brief, for some clusters of hosts providing a particular service, they are all basically "equal", and depooling any set of N hosts requires the same effort and analysis as any other random set of N hosts. In those cases moving the hosts on a rack-by-rack basis works just fine, it doesn't make a difference to the service running on them what order things are depooled. Other services, however, are not like that. For instance if specific hosts in the cluster are performing certain duties (master nodes etc), then moving the hosts in an effectively random order (rack-by-rack) can be difficult to orchestrate for the service owners. The database hosts maintained by our Data Platform were particularly tricky to get moved the last time. The best way to proceed would therefore seem to be: # Move LVS hosts as they need to connect to the new per-rack vlans before we start # Move hosts for services that may require special attention on a host-by-host basis (i.e. db's) # Move remaining hosts on a rack-by-rack basis **Planning** We probably need to set up some meetings with service owners who maintain hosts that fit into category two. To begin with I think we need to speak to: * Traffic (to discuss LVS plan) * Data Platform (to discuss DBs) * Data Persistence (swift, kafka etc) * Service Ops (most of these hosts can be done rack-by-rack) * Search (again I think can be done rack-by-rack) * Others TBC Once we've identified any sets of hosts that need special treatment we can create sub-tasks to plan the move of those hosts. We probably also need to work on some automation to move specific hosts - rather than an entire rack - from one switch to another. Lastly we need to create the per-rack move tasks for the remaining hosts, set up a schedule for the moves, and work with all the SREs to get the work completed.

Now that {T364095} is effectively completed we can begin the process of moving existing hosts from the old ASW switches in codfw rows C & D to the new Leaf switches there. **Hosts** Hosts on the following vlans need to be moved (or see [[ https://phabricator.wikimedia.org/P66915 | full list ]]): |Vlan|Hosts| |-----|-------| |private1-c-codfw|P66913| |public1-c-codfw|P66911| |private1-d-codfw|P66914| |public1-d-codfw|P66912| **Order of moves** We probably need to approach this in several phases to ensure it goes smoothly and minimise the disruption to other teams and live services during the work. One lesson from T355544, when we moved the servers from the old switches to new in rows A & B, is that for certain clusters of hosts it is a lot more difficult to schedule than others. In brief, for some clusters of hosts providing a particular service, they are all basically "equal", and depooling any set of N hosts requires the same effort and analysis as any other random set of N hosts. In those cases moving the hosts on a rack-by-rack basis works just fine, it doesn't make a difference to the service running on them what order things are depooled. Other services, however, are not like that. For instance if specific hosts in the cluster are performing certain duties (master nodes etc), then moving the hosts in an effectively random order (rack-by-rack) can be difficult to orchestrate for the service owners. The database hosts maintained by our Data Platform were particularly tricky to get moved the last time. The best way to proceed would therefore seem to be: # Move LVS hosts as they need to connect to the new per-rack vlans before we start # Move hosts for services that may require special attention on a host-by-host basis (i.e. db's) # Move remaining hosts on a rack-by-rack basis **Planning** We probably need to set up some meetings with service owners who maintain hosts that fit into category two. To begin with I think we need to speak to: * Traffic (to discuss LVS plan) * Data Platform (to discuss DBs) * Data Persistence (swift, kafka etc) * Service Ops (most of these hosts can be done rack-by-rack) * Search (again I think can be done rack-by-rack) * Others TBC Once we've identified any sets of hosts that need special treatment we can create sub-tasks to plan the move of those hosts. We probably also need to work on some automation to move specific hosts - rather than an entire rack - from one switch to another. Lastly we need to create the per-rack move tasks for the remaining hosts, set up a schedule for the moves, and work with all the SREs to get the work completed.