**Provisionally planned for Thurs July 29th at 15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST)**
Netops plan to adjust the buffer memory configuration for all switches in Eqiad Row A, to address tail drops observed on some of the devices, which is causing throughput issues.
This is an intrusive change, and will bring **all traffic on the row to a complete stop** for a short time while the switches reconfigure themselves.
All services should have row redundancy, but we may want to take some pro-active steps in advance to de-pool servers / make things go smoothly.
The exact duration of the impact is unknown at this time - we will have a much better sense after performing the same action on other rows prior to this change, and will update this task with those results. Best estimate is it will be in the order of seconds, certainly no longer than a minute, but we should plan for up to a 5-minute interruption, and be aware as always that there is a small potential something will go wrong and cause a longer disturbance.
The complete list of servers in this row can be found here:
https://netbox.wikimedia.org/dcim/devices/?q=&rack_group_id=5&status=active&role=server
Summarizing the physical servers / types:
|Server Name / Prefix |Count|Relevant Team |Action Required |Action Status|
|----------------------|-----|----------------------------------------------------------------|----------------|-------------|
|mw |28 |Service Operations |N|N/A|
|db |23 |Data Persistence | Y | |
|an-worker |14 |Analytics SREs || |
|ms-be |10 |Data Persistence (Media Storage) |N|N/A|
|elastic |9 |Search Platform SREs || |
|wtp |6 |Service Operations |N|N/A|
|analytics |5 |Analytics SREs || |
|es |5 |Data Persistence || |
|mc |5 |Service Operations || |
|cp |4 |Traffic |depool the individual hosts with the `depool` command| |
|ganeti |4 |Infrastructure Foundations |N|N/A|
|kubernetes |4 |Service Operations || |
|restbase |4 |PET @hnowlan || |
|clouddb |3 |WMCS and Analytics, with support from DBAs || |
|wdqs |3 |Search Platform SREs || |
|an-presto |2 |Analytics SREs || |
|aqs |2 |Analytics SREs || |
|cloudelastic |2 |WMCS || |
|dbproxy |2 |Data Persistence || |
|druid / an-druid |3 |Analytics SREs || |
|kafka-jumbo |2 |Analytics SREs || |
|ms-fe |2 |Data Persistence (Media Storage) |N|N/A|
|ores |2 |Machine Learning SREs || |
|stat |2 |Analytics SREs || |
|an-master1001 |1 |Analytics SREs || |
|an-test-master1001 |1 | Analytics SREs || |
|an-test-worker1001 |1 | Analytics SREs || |
|cloudcontrol1003 |1 |WMCS |N|N/A|
|cloudmetrics1002 |1 |WMCS |N|N/A|
|cloudservices1004 |1 |WMCS |N|N/A|
|conf1004 |1 |Service Operations || |
|contint1001 |1 |Service Operations || |
|dbprov1001 |1 |Data Persistence |N|N/A|
|dbstore1003 |1 |Analytics SREs & Data Persistence || |
|dns1001 |1 |Traffic || |
|htmldumper1001 |1 | || |
|kafka-main1001 |1 |Analytics || |
|krb1001 |1 |Infrastructure Foundations & Analytics SREs || |
|kubestage1001 |1 |Service Operations || |
|labstore1006 |1 |WMCS |Y| |
|logstash1010 |1 |Observability |N|N/A|
|lvs1013 |1 |Traffic |Failover to secondary (lvs1016 in row D) by stopping pybal with puppet disabled| |
|maps1001 |1 | PET @hnowlan |N|N/A|
|mc-gp1001 |1 |Service Operations || |
|ms-backup1001 |1 |Data Persistence ? |N|N/A|
|netmon1002 |1 |Infrastructure Foundations || |
|pc1007 |1 |SRE Data Persistence (DBAs), with support from Platform and Performance|| |
|pki1001 |1 |Infrastructure Foundations |N|N/A |
|prometheus1003 |1 |Observability |N|N/A|
|rdb1005 |1 |Service Operations || |
|relforge1003 |1 |Search Platform SREs || |
|restbase-dev1004 |1 |Platform Engineering |N|N/A|
|sessionstore1001 |1 |Service Operations || |
|sodium |1 |Infrastructure Foundations ?? |N|N/A |
|thanos-be1001 |1 |Observability |N|N/A|
|thanos-fe1001 |1 |Observability |N|N/A|
And the list of VMs running on Ganeti hosts in row A:
|VM Name |Ganeti Host |Team |Action Required |Action Status|
|--------------------|----------------|--------------------------------------------------------------|-----------------|-------------|
|apt1001 |ganeti1007 |Infrastructure Foundations |N|N/A|
|archiva1002 |ganeti1005 |Analytics SREs |N|N/A|
|d-i-test |ganeti1005 |Infrastructure Foundations |N|N/A|
|doc1001 |ganeti1005 |Service Operations || |
|gitlab1001 |ganeti1007 |Service Operations || |
|grafana1002 |ganeti1007 |Observability |Y|To be failed over ahead of maint|
|idp1001 |ganeti1005 |Infrastructure Foundations |Y (ensure that 2001 is the active)|idp2001 is currently the active host |
|install1003 |ganeti1006 |Infrastructure Foundations |N|N/A|
|kafkamon1001 |ganeti1006 |Analytics SREs & Infrastructure Foundations || |
|kafkamon1002 |ganeti1005 |Analytics SREs & Infrastructure Foundations || |
|kubemaster1001 |ganeti1005 |Service Operations ??/ || |
|kubernetes1005 |ganeti1006 |Service Operations || |
|kubestagetcd1004 |ganeti1005 |Service Operations || |
|kubetcd1005 |ganeti1008 |Service Operations || |
|ldap-corp1001 |ganeti1007 |Infrastructure Foundations |Y| |
|lists1001 |ganeti1007 | Service Operations and @ladsgroup |Will announce maintenance window a few days before| |
|logstash1007 |ganeti1006 |Observability |N|N/A|
|logstash1008 |ganeti1005 |Observability |N|N/A|
|logstash1023 |ganeti1005 |Observability |N|N/A|
|logstash1024 |ganeti1005 |Observability |N|N/A|
|moscovium |ganeti1006 | |N|N/A|
|mwdebug1002 |ganeti1006 |Service Operations || |
|ncredir1002 |ganeti1005 | Traffic |N|N/A|
|netbox1001 |ganeti1005 |Infrastructure Foundations |N|N/A|
|netboxdb1001 |ganeti1005 |Infrastructure Foundations |N|N/A|
|orespoolcounter1003 |ganeti1005 |Machine Learning SREs || |
|people1003 |ganeti1008 |Service Operations || |
|planet1002 |ganeti1006 |Service Operations || |
|poolcounter1004 |ganeti1005 |Service Operations || |
|registry1003 |ganeti1005 |Service Operations || |
|urldownloader1001 |ganeti1007 |Infrastructure Foundations |Y| |
|webperf1001 |ganeti1007 |Performance & Service Operations || |
|webperf1002 |ganeti1006 |Performance & Service Operations || |
I have listed the teams, and subscribed relevant individuals to this task, based on the server names and info here: https://wikitech.wikimedia.org/wiki/SRE/Infrastructure_naming_conventions. Don't hesitate to add people I could have missed, or remove yourself from the task if you do not need to be involved.
(WARNING) **Kindly update above tables if action needs to be taken for any servers/VMs. Please also list the current status of action if required, and set status to 'Complete' once work has been done.**
###### Days Before:
- Prepare config changes (netops)
###### 1h Before Window:
- Confirm switches are in a healthy state (snapshot MAC and ARP tables, port status, buffer usage) (netops)
- Warn people of the upcoming maintenance (netops)
###### After The Change
- Confirm switches are in a healthy state (snapshot MAC and ARP tables, port status, buffer usage, validate against prior values). (netops)