**Provisionally planned for Thurs July 22nd at 15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST)**
Netops plan to adjust the buffer memory configuration for all switches in Eqiad Row C, to address tail drops observed on some of the devices, which is causing throughput issues.
This is an intrusive change, and will bring **all traffic on the row to a complete stop** for a short time while the switches reconfigure themselves.
All services should have row redundancy, but we may want to take some pro-active steps in advance to de-pool servers / make things go smoothly.
The exact duration of the impact is unknown at this time - we hope to be able to test on some real switches before the date and get a firm indication. Best estimate is it will be in the order of seconds, certainly no longer than a minute, but we should plan for up to a 5-minute interruption, and be aware as always that there is a small potential something will go wrong and cause a longer disturbance.
The complete list of servers in this row can be found here:
https://netbox.wikimedia.org/dcim/devices/?q=&rack_group_id=7&status=active&role=server
Summary of hosts by type here:
|Server Name / Prefix |Count|Relevant Team |Action Required |Action Status|
|----------------------|-----|----------------------------------------------------------------|----------------|-------------|
|mw |39 |Service Operations || |
|db |23 |Data Persistence |N|N/A|
|an-worker |16 |Analytics SREs || |
|elastic |9 |Search Platform SREs || |
|ms-be |7 |Data Persistence (Media Storage) |N| |
|wtp |6 |Service Operations || |
|analytics |5 |Analytics SREs || |
|mc |5 |Service Operations || |
|cp |4 |Traffic || |
|dbproxy |4 |Data Persistence |dbproxy1018 and dbproxy1019 owned by #cloud-services-team, dbproxy1020 requires action after row D is done, dbroxy1021 doesn't | |
|ganeti |4 |Infrastructure Foundations || |
|es |3 |Data Persistence |N|N/A|
|frdb |3 |Fundraising Tech |N| |
|kafka-jumbo |3 |Analytics SREs & Infrastructure Foundations || |
|kubernetes |3 |Service Operations || |
|clouddb |2 |WMCS, with support from DBAs |Y| |
|labstore |2 |WMCS |Y| |
|ms-fe |2 |Data Persistence (Media Storage) |N| |
|ores |2 |Machine Learning SREs || |
|wdqs |2 |Search Platform SREs || |
|alert1001 |1 |Observability |Y|To be switched over ahead of maint|
|an-conf1002 |1 |Analytics || |
|an-druid1002 |1 |Analytics || |
|an-test-master1002 |1 |Analytics || |
|an-test-worker1002 |1 |Analytics || |
|aqs1005 |1 |Analytics SREs || |
|backup1002 |1 |Data Persistence || |
|cloudcontrol1005 |1 |WMCS || |
|cloudelastic1003 |1 |WMCS || |
|cloudmetrics1001 |1 |WMCS |Y| |
|cumin1001 |1 |Infrastructure Foundations || |
|dbprov1003 |1 |Data Persistence || |
|dbstore1005 |1 |Analytics SREs & Data Persistence || |
|druid1002 |1 | Analytics || |
|dumpsdata1003 |1 |Service Operations & Platform Engineering || |
|kafka-main1003 |1 |Analytics || |
|lvs1015 |1 |Traffic || |
|maps1003 |1 | || |
|mc-gp1002 |1 |Service Operations || |
|ms-backup1002 |1 |Data Persistence ?? || |
|mwlog1002 |1 |Service Operations |N| |
|pc1009 |1 |SRE Data Persistence (DBAs), with support from Platform and Performance|N|N/A|
|sessionstore1002 |1 |Service Operations || |
|thanos-be1003 |1 |Observability |N| |
|thanos-fe1003 |1 |Observability |N| |
VMs on this row are as follows:
|VM Name |Ganeti Host |Team |Action Required |Action Status|
|--------------------|----------------|--------------------------------------------------------------|-----------------|-------------|
|acmechief-test1001 |ganeti1009 |Traffic || |
|acmechief1001 |ganeti1009 |Traffic || |
|an-airflow1001 |ganeti1010 |Analytics SREs || |
|an-tool1005 |ganeti1009 |Analytics SREs || |
|an-tool1007 |ganeti1010 |Analytics SREs || |
|doc1002 |ganeti1009 | || |
|doh1001 |ganeti1011 |Traffic || |
|etherpad1002 |ganeti1009 |Service Operations || |
|flowspec1001 |ganeti1009 |Infrastructure Foundations |N|N/A|
|idp-test1001 |ganeti1010 |Infrastructure Foundations |N|N/A|
|kubemaster1002 |ganeti1010 |Service Operations || |
|kubernetes1006 |ganeti1009 |Service Operations || |
|kubestagetcd1006 |ganeti1012 |Service Operations || |
|kubetcd1004 |ganeti1010 |Service Operations || |
|logstash1009 |ganeti1010 |Observability |N| |
|logstash1025 |ganeti1009 |Observability |N| |
|matomo1002 |ganeti1009 |Analytics || |
|miscweb1002 |ganeti1009 |Service Operations |N|N/A|
|ml-etcd1002 |ganeti1012 |ML team || |
|mwdebug1001 |ganeti1010 |Service Operations || |
|mx1001 |ganeti1009 |Infrastructure Foundations |N|N/A|
|ncredir1001 |ganeti1009 |Traffic || |
|netflow1001 |ganeti1012 |Infrastructure Foundations |N|N/A|
|orespoolcounter1004 |ganeti1010 |Machine Learning SREs || |
|ping1001 |ganeti1009 |Infrastructure Foundations |Y| |
|poolcounter1005 |ganeti1010 |Service Operations |Y| |
|puppetboard1001 |ganeti1010 |Infrastructure Foundations |N|N/A |
|puppetdb1002 |ganeti1012 |Infrastructure Foundations |Y (disable Puppet fleet-wide during maintenance)| |
|registry1004 |ganeti1009 |Service Operations || |
|rpki1001 |ganeti1009 |Infrastructure Foundations |N|N/A|
|seaborgium |ganeti1010 |Infrastructure Foundations |N|N/A|
|urldownloader1002 |ganeti1010 |Infrastructure Foundations |Y| |
I have listed the teams, and subscribed relevant individuals to this task, based mostly on the server names and info here: https://wikitech.wikimedia.org/wiki/SRE/Infrastructure_naming_conventions. Don't hesitate to add people I could have missed, or remove yourself from the task if you do not need to be involved.
(WARNING) **Kindly update above tables if action needs to be taken for any servers/VMs. Please also list those actions in the below timeline (when known), along with who will take care of them.**
###### Days Before:
- Prepare config changes (netops)
- Alter process control timing for long running jobs (frtech)
- Set maintenance window for fundraising banners (frtech)
###### 1h Before Window:
- Confirm switches are in a healthy state (snapshot MAC and ARP tables, port status, buffer usage) (netops)
- Warn people of the upcoming maintenance (netops)
- Depool ping1001 ([[ https://wikitech.wikimedia.org/wiki/Ping_offload#Temporarily_stop_the_ICMP_echo_redirect | doc ]]) (netops)
- Set maintenance mode for fundraising servers (most likely 5 mins before window) (frtech)
###### After The Change
- Confirm switches are in a healthy state (snapshot MAC and ARP tables, port status, buffer usage, validate against prior values). (netops)
- Unset maintenance mode for fundraising servers and return process control jobs to normal schedule. (frtech)