Depooling of GeoDNS and active/active service in the source datacenter (eqiad).
sudo cookbook sre.dns.admin depool eqiad
sudo cookbook sre.discovery.datacenter status all
Note status in comment.
sudo cookbook sre.discovery.datacenter --task-id T435443 depool --all --reason "Datacenter services switchover" eqiad
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | SLyngshede-WMF | T433363 🏖️ Southward Datacenter Switchover (Sept. 2026) | |||
| Declined | SLyngshede-WMF | T435443 Day 1 - Source datacenter depooling (Tue. Sept. 22nd - 2026) |
Change #1343920 had a related patch set uploaded (by Slyngshede; author: Slyngshede):
[operations/puppet@production] Service::catalog exclude rest-gateway and Sophroid from DC switch
Change #1343921 had a related patch set uploaded (by Slyngshede; author: Slyngshede):
[operations/puppet@production] service::catalog: exclude sessionstore from DC switchover
slyngshede@cumin1004:~$ sudo cookbook sre.discovery.datacenter status all Service Type eqiad codfw ================================================================= apertium Active/Active pooled pooled apt Active/Passive pooled apus Active/Active pooled pooled citoid Active/Active pooled pooled config-master Active/Active pooled pooled cxserver Active/Active pooled pooled device-analytics Active/Active pooled pooled docker-registry Active/Passive pooled echostore Active/Active pooled pooled eventgate-analytics Active/Active pooled pooled eventgate-analytics-external Active/Active pooled pooled eventgate-logging-external Active/Active pooled pooled eventgate-main Active/Active pooled pooled eventstreams Active/Active pooled pooled helm-charts Active/Active pooled pooled inference Active/Active pooled pooled k8s-ingress-aux-ro Active/Active pooled pooled k8s-ingress-aux-rw Active/Passive pooled k8s-ingress-dse Active/Passive pooled k8s-ingress-dse-aa Active/Active pooled pooled k8s-ingress-ml-serve Active/Active pooled pooled k8s-ingress-ml-staging Active/Passive pooled k8s-ingress-staging Active/Passive pooled k8s-ingress-wikikube-ro Active/Active pooled pooled k8s-ingress-wikikube-rw Active/Passive pooled kartotherian Active/Active pooled pooled linkrecommendation Active/Active pooled pooled logstash Active/Passive pooled mathoid Active/Active pooled pooled mobileapps Active/Active pooled pooled mw-api-ext Active/Passive pooled mw-api-ext-next Active/Passive pooled mw-api-ext-next-ro Active/Active pooled pooled mw-api-ext-ro Active/Active pooled pooled mw-api-int Active/Passive pooled mw-api-int-ro Active/Active pooled pooled mw-jobrunner Active/Passive pooled mw-web Active/Passive pooled mw-web-next Active/Passive pooled mw-web-next-ro Active/Active pooled pooled mw-web-ro Active/Active pooled pooled mwdebug Active/Active pooled mwdebug-next Active/Active pooled netbox Active/Passive pooled pki Active/Active pooled pooled proton Active/Active pooled pooled puppetboard Active/Active pooled pooled puppetdb-api Active/Active pooled pooled push-notifications Active/Active pooled pooled recommendation-api Active/Active pooled pooled releases Active/Active pooled pooled rest-gateway Active/Passive pooled rest-gateway-ro Active/Active pooled pooled restbase Active/Active pooled pooled restbase-async Active/Active pooled pooled schema Active/Active pooled pooled search Active/Active pooled pooled search-omega Active/Active pooled pooled search-psi Active/Active pooled pooled sessionstore Active/Active pooled pooled shellbox Active/Active pooled pooled shellbox-constraints Active/Active pooled pooled shellbox-media Active/Active pooled pooled shellbox-syntaxhighlight Active/Active pooled pooled shellbox-timeline Active/Active pooled pooled shellbox-video Active/Active pooled pooled sophroid Active/Passive pooled swift Active/Active pooled pooled tegola-vector-tiles Active/Active pooled pooled termbox Active/Active pooled pooled thanos-query Active/Active pooled pooled thanos-swift Active/Active pooled pooled thanos-web Active/Active pooled pooled toolhub Active/Passive pooled urldownloader Active/Active pooled pooled wcqs Active/Active pooled pooled wdqs-internal-main Active/Active pooled pooled wdqs-internal-scholarly Active/Active pooled pooled wdqs-main Active/Active pooled pooled wdqs-scholarly Active/Active pooled pooled wikifeeds Active/Active pooled pooled zotero Active/Active pooled pooled
Change #1343921 merged by Slyngshede:
[operations/puppet@production] service::catalog: exclude sessionstore from DC switchover
Change #1344000 had a related patch set uploaded (by Slyngshede; author: Slyngshede):
[operations/puppet@production] service::catalog: exclude k8s-ingress-aux
Change #1344000 merged by Slyngshede:
[operations/puppet@production] service::catalog: exclude k8s-ingress-aux
Updated status after patches:
Service Type eqiad codfw ================================================================= apertium Active/Active pooled pooled apt Active/Passive pooled apus Active/Active pooled pooled citoid Active/Active pooled pooled config-master Active/Active pooled pooled cxserver Active/Active pooled pooled device-analytics Active/Active pooled pooled docker-registry Active/Passive pooled echostore Active/Active pooled pooled eventgate-analytics Active/Active pooled pooled eventgate-analytics-external Active/Active pooled pooled eventgate-logging-external Active/Active pooled pooled eventgate-main Active/Active pooled pooled eventstreams Active/Active pooled pooled helm-charts Active/Active pooled pooled inference Active/Active pooled pooled k8s-ingress-aux-ro Active/Active pooled pooled k8s-ingress-aux-rw Active/Passive pooled k8s-ingress-dse Active/Passive pooled k8s-ingress-dse-aa Active/Active pooled pooled k8s-ingress-ml-serve Active/Active pooled pooled k8s-ingress-ml-staging Active/Passive pooled k8s-ingress-staging Active/Passive pooled k8s-ingress-wikikube-ro Active/Active pooled pooled k8s-ingress-wikikube-rw Active/Passive pooled kartotherian Active/Active pooled pooled linkrecommendation Active/Active pooled pooled logstash Active/Passive pooled mathoid Active/Active pooled pooled mobileapps Active/Active pooled pooled mw-api-ext Active/Passive pooled mw-api-ext-next Active/Passive pooled mw-api-ext-next-ro Active/Active pooled pooled mw-api-ext-ro Active/Active pooled pooled mw-api-int Active/Passive pooled mw-api-int-ro Active/Active pooled pooled mw-jobrunner Active/Passive pooled mw-web Active/Passive pooled mw-web-next Active/Passive pooled mw-web-next-ro Active/Active pooled pooled mw-web-ro Active/Active pooled pooled mwdebug Active/Active pooled mwdebug-next Active/Active pooled netbox Active/Passive pooled pki Active/Active pooled pooled proton Active/Active pooled pooled puppetboard Active/Active pooled pooled puppetdb-api Active/Active pooled pooled push-notifications Active/Active pooled pooled recommendation-api Active/Active pooled pooled releases Active/Active pooled pooled rest-gateway Active/Passive pooled rest-gateway-ro Active/Active pooled pooled restbase Active/Active pooled pooled restbase-async Active/Active pooled pooled schema Active/Active pooled pooled search Active/Active pooled pooled search-omega Active/Active pooled pooled search-psi Active/Active pooled pooled sessionstore Active/Active pooled pooled shellbox Active/Active pooled pooled shellbox-constraints Active/Active pooled pooled shellbox-media Active/Active pooled pooled shellbox-syntaxhighlight Active/Active pooled pooled shellbox-timeline Active/Active pooled pooled shellbox-video Active/Active pooled pooled sophroid Active/Passive pooled swift Active/Active pooled pooled tegola-vector-tiles Active/Active pooled pooled termbox Active/Active pooled pooled thanos-query Active/Active pooled pooled thanos-swift Active/Active pooled pooled thanos-web Active/Active pooled pooled toolhub Active/Passive pooled urldownloader Active/Active pooled pooled wcqs Active/Active pooled pooled wdqs-internal-main Active/Active pooled pooled wdqs-internal-scholarly Active/Active pooled pooled wdqs-main Active/Active pooled pooled wdqs-scholarly Active/Active pooled pooled wikifeeds Active/Active pooled pooled zotero Active/Active pooled pooled slyngshede@cumin1004:~$
slyngshede@cumin1004 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T435443 started.
Mentioned in SAL (#wikimedia-operations) [2026-09-22T14:55:30Z] <slyngshede@cumin1004> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T435443
slyngshede@cumin1004 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T435443 completed.
Mentioned in SAL (#wikimedia-operations) [2026-09-22T15:22:19Z] <slyngshede@cumin1004> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - T435443
Update: PHP-FPM saturation in codfw, following the traffic & services switchover, eqiad has been repooled for mw-web-ro traffic.
Edit: Followed by repooling mw-api-ext-ro and mw-api-int-ro in codfw as well.
Mentioned in SAL (#wikimedia-operations) [2026-09-23T00:46:43Z] <ryankemper> [WDQS] T435443 Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture
Change #1344974 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] wdqs: exclude public main from DC switchover