Many of the required fixes have yet to roll out, but the situation appears to have worsened since last switchover attempt.

@Aklapper It works perfectly, thank you :-)

The offboarding script failed to remove a user from "acl_support_and_safety_policy_admins". The script authenticates as "offboarding".

@MLechvien-WMF I can certainly take a look :-)

It's fine... Phabricator has a million settings for policies, yet I can't make a task "blocking".

@Novem_Linguae Thanks, fixed. It's in the deployment calendar anyway and Move Comms will be announcing things, so hardly a secret :-)

Script no longer exists.

No longer in use

check_conntrack.py no longer exists.

You should be able to update the information yourself. First ensure that you're signed out from your Wikimedia Global Account (or signed in with the correct account). Then click the "Refresh" link below the "Wikimedia Global Account (SUL)" box.

@Trizek-WMF Hi, first day of switchover revealed a capacity issue. We're cancelling the rest of the planned maintenance to avoid any risk, while we work on resolving the issue.

Updated status after patches:

We'll be skipping k8s-ingress-aux-rw due to doubts

slyngshede@cumin1004:~$ sudo cookbook sre.discovery.datacenter status all

Service Type eqiad codfw

apertium Active/Active pooled pooled
apt Active/Passive pooled
apus Active/Active pooled pooled
citoid Active/Active pooled pooled
config-master Active/Active pooled pooled
cxserver Active/Active pooled pooled
device-analytics Active/Active pooled pooled
docker-registry Active/Passive pooled
echostore Active/Active pooled pooled
eventgate-analytics Active/Active pooled pooled
eventgate-analytics-external Active/Active pooled pooled
eventgate-logging-external Active/Active pooled pooled
eventgate-main Active/Active pooled pooled
eventstreams Active/Active pooled pooled
helm-charts Active/Active pooled pooled
inference Active/Active pooled pooled
k8s-ingress-aux-ro Active/Active pooled pooled
k8s-ingress-aux-rw Active/Passive pooled
k8s-ingress-dse Active/Passive pooled
k8s-ingress-dse-aa Active/Active pooled pooled
k8s-ingress-ml-serve Active/Active pooled pooled
k8s-ingress-ml-staging Active/Passive pooled
k8s-ingress-staging Active/Passive pooled
k8s-ingress-wikikube-ro Active/Active pooled pooled
k8s-ingress-wikikube-rw Active/Passive pooled
kartotherian Active/Active pooled pooled
linkrecommendation Active/Active pooled pooled
logstash Active/Passive pooled
mathoid Active/Active pooled pooled
mobileapps Active/Active pooled pooled
mw-api-ext Active/Passive pooled
mw-api-ext-next Active/Passive pooled
mw-api-ext-next-ro Active/Active pooled pooled
mw-api-ext-ro Active/Active pooled pooled
mw-api-int Active/Passive pooled
mw-api-int-ro Active/Active pooled pooled
mw-jobrunner Active/Passive pooled
mw-web Active/Passive pooled
mw-web-next Active/Passive pooled
mw-web-next-ro Active/Active pooled pooled
mw-web-ro Active/Active pooled pooled
mwdebug Active/Active pooled
mwdebug-next Active/Active pooled
netbox Active/Passive pooled
pki Active/Active pooled pooled
proton Active/Active pooled pooled
puppetboard Active/Active pooled pooled
puppetdb-api Active/Active pooled pooled
push-notifications Active/Active pooled pooled
recommendation-api Active/Active pooled pooled
releases Active/Active pooled pooled
rest-gateway Active/Passive pooled
rest-gateway-ro Active/Active pooled pooled
restbase Active/Active pooled pooled
restbase-async Active/Active pooled pooled
schema Active/Active pooled pooled
search Active/Active pooled pooled
search-omega Active/Active pooled pooled
search-psi Active/Active pooled pooled
sessionstore Active/Active pooled pooled
shellbox Active/Active pooled pooled
shellbox-constraints Active/Active pooled pooled
shellbox-media Active/Active pooled pooled
shellbox-syntaxhighlight Active/Active pooled pooled
shellbox-timeline Active/Active pooled pooled
shellbox-video Active/Active pooled pooled
sophroid Active/Passive pooled
swift Active/Active pooled pooled
tegola-vector-tiles Active/Active pooled pooled
termbox Active/Active pooled pooled
thanos-query Active/Active pooled pooled
thanos-swift Active/Active pooled pooled
thanos-web Active/Active pooled pooled
toolhub Active/Passive pooled
urldownloader Active/Active pooled pooled
wcqs Active/Active pooled pooled
wdqs-internal-main Active/Active pooled pooled
wdqs-internal-scholarly Active/Active pooled pooled
wdqs-main Active/Active pooled pooled
wdqs-scholarly Active/Active pooled pooled
wikifeeds Active/Active pooled pooled
zotero Active/Active pooled pooled

@Trizek-WMF I'll be handling the information. I can keep you updated on this task, unless you prefer some other channel.

Traffic is moving to ULSFO: https://grafana.wikimedia.org/d/000000093/cdn-frontend-network?from=now-30m&to=now&timezone=utc&var-site=eqsin&var-site=ulsfo&refresh=auto