Planned for Tues July 20th at 15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST)
UPDATE: Works completed without incident at scheduled time. Resolving task.
Netops plan to adjust the buffer memory configuration for all switches in Eqiad Row D, to address tail drops observed on some of the devices, which are causing throughput issues.
This is an intrusive change, and will bring all traffic on the row to a complete stop for a short time while the switches reconfigure themselves.
All services should have row redundancy, but we may want to take some pro-active steps in advance to de-pool servers / make things go smoothly.
The exact duration of the impact is unknown at this time - we hope to be able to test on some real switches before the date and get a firm indication. Best estimate is it will be in the order of seconds, certainly no longer than a minute, but we should plan for up to a 5-minute interruption, and be aware as always that there is a small potential something will go wrong and cause a longer disturbance.
The complete list of servers in this row can be found here:
https://netbox.wikimedia.org/dcim/devices/?q=&rack_group_id=8&status=active&role=server
Summary of server types in the row as follows:
| Server Name / Prefix | Count | Relevant Team | Action Required | Action Status |
|---|---|---|---|---|
| mw | 36 | Service Operations | N | N/A |
| db | 20 | Data Persistence | N | N/A |
| an-worker | 14 | Analytics SREs | N | N/A |
| ms-be | 9 | Data Persistence (Media Storage) | N | N/A |
| elastic | 8 | Search Platform SREs | N | N/A |
| wtp | 6 | Service Operations | N | N/A |
| analytics | 4 | Analytics SREs | N | N/A |
| cp | 4 | Traffic | depool the individual hosts with the depool command | complete |
| ganeti | 4 | Infrastructure Foundations | N | N/A |
| mc | 4 | Service Operations | N | N/A |
| restbase | 4 | Service Operations | N | N/A |
| druid | 3 | Analytics | N | N/A |
| es | 3 | Data Persistence | N | N/A |
| kafka-jumbo | 3 | Analytics SREs & Infrastructure Foundations | N | N/A |
| kubernetes | 3 | Service Operations | N | N/A |
| ores | 3 | Machine Learning SREs | N | N/A |
| an-presto | 2 | Analytics | N | N/A |
| aqs | 2 | Analytics SREs | N | N/A |
| clouddb | 2 | WMCS, with support from DBAs | Y | Done |
| cloudstore | 2 | WMCS | N | N/A |
| dbproxy | 2 | Data Persistence | N | N/A |
| kafka-main | 2 | Analytics | Y - Keith to depool in advance | Complete |
| rdb | 2 | Service Operations | N - But we want to do quick check afterwards | N/A |
| stat | 2 | Analytics SREs | N | N/A |
| thumbor | 2 | Service Operations (& Performance) | N | N/A |
| wdqs | 2 | Search Platform SREs | N | N/A |
| an-conf1003 | 1 | Analytics | N | N/A |
| an-test-coord1001 | 1 | Analytics | N | N/A |
| an-test-worker1003 | 1 | Analytics | N | N/A |
| backup1001 | 1 | Data Persistence | Heads up to Jaime before | Completed |
| bast1003 | 1 | Infrastructure Foundations | Tell people in advance to use a different bastion | Complete |
| centrallog1001 | 1 | Observability | N | N/A |
| cloudelastic1004 | 1 | Search Platform | N | N/A |
| conf1006 | 1 | Service Operations | Y - Advise traffic team afterwards so they can restart any PyBal instances connected to this | To be done after |
| dbstore1007 | 1 | Analytics SREs & Data Persistence | N | N/A |
| dns1002 | 1 | Traffic | Y - depool ahead of change | Complete |
| dumpsdata1002 | 1 | Service Operations & Platform Engineering | N | N/A |
| flerovium | 1 | Analytics | N | N/A |
| labstore1007 | 1 | WMCS | N | N/A |
| labweb1002 | 1 | WMCS | N | N/A |
| logstash1012 | 1 | Observability | N | N/A |
| lvs1016 | 1 | Traffic | N | N/A |
| maps1004 | 1 | N | N/A | |
| mc-gp1003 | 1 | Service Operations | N | N/A |
| pc1010 | 1 | SRE Data Persistence (DBAs), with support from Platform and Performance | N | N/A |
| puppetmaster1002 | 1 | Infrastructure Foundations | Disable puppet fleet wide | Complete |
| sessionstore1003 | 1 | Service Operations | N | N/A |
| snapshot1009 | 1 | Service Operations & Platform Engineering | Y | Complete |
| thanos-be1004 | 1 | Observability | N | N/A |
| thorium | 1 | Analytics | N | N/A |
The VMs on this row are as follows:
| VM Name | Ganeti Host | Team | Action Required | Action Status |
|---|---|---|---|---|
| an-airflow1003 | ganeti1021 | Analytics SREs | N | N/A |
| an-test-druid1001 | ganeti1020 | Analytics | N | N/A |
| an-test-presto1001 | ganeti1020 | Analytics | N | N/A |
| aphlict1001 | ganeti1020 | Service Operations | N | N/A |
| chartmuseum1001 | ganeti1019 | Service Operations | N | N/A |
| cuminunpriv1001 | ganeti1020 | Infrastructure Foundations | N | N/A |
| dbmonitor1002 | ganeti1019 | Data Persistence | N | N/A |
| dborch1001 | ganeti1019 | Data Persistence | N | N/A |
| doh1002 | ganeti1022 | Traffic | N | N/A |
| eventlog1003 | ganeti1022 | Analytics SREs | N | N/A |
| irc1001 | ganeti1019 | Infra Foundations | Y (failover to irc2001) | Complete |
| kubernetes1016 | ganeti1021 | Service Operations | N | N/A |
| ldap-replica1004 | ganeti1021 | Infrastructure Foundations | Y | Complete |
| logstash1030 | ganeti1020 | Observability | N | N/A |
| logstash1031 | ganeti1019 | Observability | N | N/A |
| ml-etcd1003 | ganeti1019 | ML team | N | N/A |
| ml-serve-ctrl1002 | ganeti1020 | ML team | N | N/A |
| puppetboard1002 | ganeti1021 | Infrastructure Foundations | N | N/A |
| releases1002 | ganeti1019 | Service Operations | N | N/A |
| schema1004 | ganeti1019 | Analytics SREs & Service Operations | N | N/A |
| search-loader1001 | ganeti1020 | Search Platform SREs | N | N/A |
| testreduce1001 | ganeti1019 | Service Operations | N | N/A |
| xhgui1001 | ganeti1020 | Performance | N | N/A |
I have listed the teams, and subscribed relevant individuals to this task, based mostly on the server names and info here: https://wikitech.wikimedia.org/wiki/SRE/Infrastructure_naming_conventions. Don't hesitate to add people I could have missed, or remove yourself from the task if you do not need to be involved.
Days Before:
- Prepare config changes (netops)
1h Before Window:
- Confirm switches are in a healthy state (snapshot MAC and ARP tables, port status, buffer usage) (netops)
- Warn people of the upcoming maintenance (netops)
After The Change
- Confirm switches are in a healthy state (snapshot MAC and ARP tables, port status, buffer usage, validate against prior values). (netops)