This is scheduled for Feb 7th - 14:00-16:00 UTC, please let us know if there is any issue with the scheduled time.
It means a 30min hard downtime for the whole row if everything goes well. Also a good opportunity to test the hosts depool mechanisms and row redundancy of services.

The list of impacted servers and teams for this row is listed below.
The actions needed is quite free form:

please write NONE if no action is needed,
the cookbook/command to run if it can be done by a 3rd party
who will be around to take care of the depool
Link to the relevant doc
etc

The two main types of actions needed are depool and monitoring downtime

NOTE: If the servers can handle a longer depool, it's preferred to depool them many hours or the day before (and mark None in the table) so there are less moving parts closer to the maintenance window.

Data Engineering

Data-Engineering

Servers	Depool action needed	Repool action needed	Status
aqs[2001-2004]	None	None

Observability

SRE Observability

Servers	Depool action needed	Repool action needed	Status
grafana2001	set downtime	none	depooled
kafka-logging2001	set downtime, stop kafka service	start kafka service, confirm kafka logging dashboard returns green	depooled
kafkamon2002	set downtime	none	depooled
logstash[2001,2023,2026,2033]	conftool 2023, drain shards 2001,2026,2033	conftool 2023, allocate shards 2001,2026,2033	all re-pooled
xhgui2001	none	none	n/a

Observability and Data Persistence

SRE Observability Data-Persistence

Servers	Depool action needed	Repool action needed	Status
thanos-fe2001	conftool depool, while making sure another thanos-fe host is pooled for service `thanos-web`	conftool pool. make sure only one thanos-fe host is pooled for `thanos-web` service

Search Platform

Discovery-Search

Servers	Depool action needed	Repool action needed	Status
elastic[2037-2040,2055-2056,2061-2062,2069,2073-2076]	None	None	Search team will depool & ban hosts from cluster one day prior to upgrade
wdqs[2003-2004,2009]	None	None	Search team will depool one day prior to upgrade

Core Platform

Platform Engineering

Servers	Depool action needed	Repool action needed	Status
maps2005
thumbor2005

WMCS

cloud-services-team

Servers	Depool action needed	Repool action needed	Status
cloudbackup2001	NONE	NONE

ServiceOps-Collab

collaboration-services

Servers	Depool action needed	Repool action needed
contint2001	NONE	NONE
doc2001	NONE	NONE
gitlab2002	NONE	NONE
planet2002	NONE	NONE

Data Persistence

Data-Persistence

Servers	Depool action needed	Repool action needed	Status
backup[2002,2004]	They are not a service, but storage. Jaime will make sure earlier in the week they are not active at the time of the maintenance.	Jaime will restart some delayed backups, if any.	no blockers
db[2094,2097,2103-2106,2121-2122,2132-2133,2136,2142,2145-2146,2153-2158,2175-2176,2183]	All MW need to be depooled, and some masters need to be switched over (misc masters do not need switchover/depooling	@Marostegui will repool everything	@Marostegui No longer masters: db2103, db2104, db2105, db2121, db2142 - the rest of masters are misc so they can be ignored - what needs to be depooled, is already depooled
dbprov2001	They are not a service, but storage. Jaime will make sure earlier in the week they are not active at the time of the maintenance.	None	no blockers
dbproxy2001	None	Reload haproxy
es[2020,2024,2026-2028]	All need to be depooled (@Marostegui will do it)	@Marostegui will repool everything	Depooled
moss-be2001	N/A	N/A	Not currently in production service
ms-be[2040,2044-2045,2051-2052,2060,2062,2066]	None	None
ms-fe2009	`sudo depool`	`sudo pool`
pc2011	To be depooled	To be repooled once it is all done	Already depooled by @Marostegui
thanos-be2001	None	None

Infrastructure Foundations

Infrastructure-Foundations

Servers	Depool action needed	Repool action needed	Status
ganeti[2023-2024,2027-2030]	None
ganeti-test[2001-2003]	None
netbox2002	None	None
netboxdb2002	None	None
pki2001	None	None	N/A
puppetdb2002	`sudo cumin 'A:codfw or A:esams or A:ulsfo' 'disable-puppet "Switch reboot: T327925"'`	`sudo cumin 'A:codfw or A:esams or A:ulsfo' 'enable-puppet "Switch reboot: T327925"'`	jbond will handle
puppetmaster[2001,2004]	`sudo cumin 'A:codfw or A:esams or A:ulsfo' 'disable-puppet "Switch reboot: T327925"'`	`sudo cumin 'A:codfw or A:esams or A:ulsfo' 'enable-puppet "Switch reboot: T327925"'`	jbond will handle
rpki2002	None
test-reimage2001	nNone
testvm[2001-2005]	None
urldownloader2001	None

Infrastructure Foundations and Observability

Infrastructure-Foundations SRE Observability

Servers	Depool action needed	Repool action needed	Status
netmon2002	None	None

Machine Learning

Machine-Learning-Team

Servers	Depool action needed	Repool action needed	Status
ml-cache2001	-	-
ml-serve[2001,2005]	-	-
ml-staging2001	-	-
ml-staging-etcd2001	-	-
ores[2001-2002]	`sudo -i depool`	`sudo -i pool`
orespoolcounter2003	-	-	-

Traffic

Servers	Depool action needed	Repool action needed	Status
acmechief2001	N/A
acmechief-test2001	N/A
authdns2001	redirect to authdns1001	the opposite	done
cp[2027-2030]	N/A	N/A
doh2001	disable puppet & stop bird.service	the oppposite	done
lvs2007	N/A	N/A
ncredir2001	N/A	N/A

ServiceOps

serviceops
Due to the large number of services potentially affected (multiple mw appservers, kubernetes workers), a global depool of a/a services was done:
sre.discovery.datacenter-route depool --reason T327925 codfw
After the maintenance, repool:
sre.discovery.datacenter-route pool --reason T327925 codfw
Depool restbase-async from eqiad:
cookbook sre.discovery.service-route --reason T327925 depool --wipe-cache eqiad restbase-async

Servers	Depool action needed	Repool action needed	Status
kafka-main2001			done
kubemaster2001			done
kubernetes[2005,2007-2008,2018-2019]			done
kubestage2001			done
kubetcd2004			done
mc[2038-2041,2055]			done
mc-gp2001			done
mw[2291-2309,2377-2411]			done
mwdebug2001			done
parse[2001-2005]			done
poolcounter2003			done
rdb2007			done
registry2003			done

Details

Subject	Repo	Branch	Lines +/-
Revert "dbbackups: Delay codfw es (db content) backups by one day"	operations/puppet	production	+0 -1
Revert "admin_state: depool codfw"	operations/dns	master	+0 -1
admin_state: depool codfw	operations/dns	master	+1 -0
dbbackups: Delay codfw es (db content) backups by one day	operations/puppet	production	+1 -0
ProductionServices.php: Depool pc2011	operations/mediawiki-config	master	+2 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T253824 planned upstream deprecation of the ssh-rsa signing algorithm (RSA with SHA-1)
Resolved	ayounsi	T254013 all network devices must run OpenSSH >= 7.2p1 but != 7.4p1
Resolved	ayounsi	T317175 Junos: resolve DNS through mgmt_junos
Resolved	ayounsi	T327862 Use mgmt_junos on all network devices
		Restricted Task
Open	None	T316539 Upgrade network devices to Junos 20+
Resolved	ayounsi	T327248 eqiad/codfw virtual-chassis upgrades
Resolved	ayounsi	T327925 codfw row A switches upgrade
Resolved	Marostegui	T327997 Switchover s1 master (db2103 -> db2112)
Resolved	Marostegui	T327998 Switchover s2 master (db2104 -> db2107)
Resolved	Marostegui	T327999 Switchover s3 master (db2105 -> db2127)
Resolved	Marostegui	T328000 Switchover s7 master (db2121 -> db2118)
Resolved	Marostegui	T328001 Switchover x2 master (db2142 -> db2144)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Eevans updated the task description. (Show Details)Jan 26 2023, 5:14 PM

herron updated the task description. (Show Details)Jan 26 2023, 6:16 PM

herron updated the task description. (Show Details)

RKemper updated the task description. (Show Details)Jan 26 2023, 8:40 PM

Marostegui updated the task description. (Show Details)Jan 27 2023, 8:52 AM

Jelto updated the task description. (Show Details)Jan 27 2023, 9:48 AM

fgiunchedi updated the task description. (Show Details)Jan 27 2023, 10:01 AM

MatthewVernon updated the task description. (Show Details)Jan 27 2023, 10:04 AM

elukey updated the task description. (Show Details)Jan 27 2023, 10:12 AM

fgiunchedi updated the task description. (Show Details)Jan 27 2023, 10:18 AM

ayounsi moved this task from Backlog to This quarter on the netops board.Jan 27 2023, 10:42 AM

KOfori subscribed.Jan 27 2023, 10:17 PM

• MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board.Jan 30 2023, 4:29 PM

• MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search.

LSobanski updated the task description. (Show Details)Jan 30 2023, 4:48 PM

LSobanski moved this task from Incoming to Consultation on the collaboration-services board.

elukey moved this task from Unsorted to Watching on the Machine-Learning-Team board.Jan 31 2023, 9:41 AM

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jan 31 2023, 4:10 PM

• EChetty moved this task from Backlog to Shared Data Infra on the Data-Engineering-Planning board.Jan 31 2023, 4:12 PM

• EChetty added a project: Shared-Data-Infrastructure.

colewhite updated the task description. (Show Details)Jan 31 2023, 7:14 PM

Clement_Goubert moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops board.Feb 1 2023, 1:03 PM

Marostegui updated the task description. (Show Details)Feb 3 2023, 8:00 AM

• MoritzMuehlenhoff updated the task description. (Show Details)Feb 3 2023, 10:55 AM

colewhite updated the task description. (Show Details)Feb 3 2023, 10:14 PM

I would suggest that instead of handling individual systems, we depool the whole datacenter from external and internal traffic for the duration of the maintenance.

That is, unless we want to verify we can survive the loss of one Row and then i'd suggest we do nothing instead.

In T327925#8587104, @Joe wrote:

I would suggest that instead of handling individual systems, we depool the whole datacenter from external and internal traffic for the duration of the maintenance.

That is, unless we want to verify we can survive the loss of one Row and then i'd suggest we do nothing instead.

+1 from my side (and I would apply this to all the future rows maintenance that is already scheduled)

Mentioned in SAL (#wikimedia-operations) [2023-02-06T07:19:13Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2020 es2024 es2026 es2027 es2028 T327925', diff saved to https://phabricator.wikimedia.org/P43586 and previous config saved to /var/cache/conftool/dbconfig/20230206-071913-root.json

Marostegui updated the task description. (Show Details)Feb 6 2023, 7:19 AM

Mentioned in SAL (#wikimedia-operations) [2023-02-06T07:30:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2094 db2097 db2103 db2104 db2105 db2106 db2121 db2122 db2132 db2133 db2136 db2142 db2145 db2146 db2153 db2154 db2155 db2156 db2157 db2158 db2175 db2176 db2183 T327925', diff saved to https://phabricator.wikimedia.org/P43587 and previous config saved to /var/cache/conftool/dbconfig/20230206-073015-root.json

Marostegui updated the task description. (Show Details)Feb 6 2023, 7:30 AM

ayounsi updated the task description. (Show Details)Feb 6 2023, 8:43 AM

• MoritzMuehlenhoff updated the task description. (Show Details)Feb 6 2023, 8:49 AM

Change 886834 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Delay codfw es (db content) backups by one day

https://gerrit.wikimedia.org/r/886834

gerritbot added a project: Patch-For-Review.Feb 6 2023, 9:42 AM

In T327925#8587186, @Marostegui wrote:

In T327925#8587104, @Joe wrote:

I would suggest that instead of handling individual systems, we depool the whole datacenter from external and internal traffic for the duration of the maintenance.

That is, unless we want to verify we can survive the loss of one Row and then i'd suggest we do nothing instead.

+1 from my side (and I would apply this to all the future rows maintenance that is already scheduled)

Yeah, let's avoid causing issues to end users and having to stress about that. Let's fully depool codfw.

• EChetty moved this task from Backlog to To be discussed on the Shared-Data-Infrastructure board.Feb 6 2023, 10:08 AM

Cool! I am going to repool the hosts then :)

Change 886834 merged by Jcrespo:

[operations/puppet@production] dbbackups: Delay codfw es (db content) backups by one day

https://gerrit.wikimedia.org/r/886834

Change 886812 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] Revert "dbbackups: Delay codfw es (db content) backups by one day"

https://gerrit.wikimedia.org/r/886812

dcaro updated the task description. (Show Details)Feb 6 2023, 12:47 PM

I am repooling all the databases since we are going to fully depool codfw for reads.

ssingh updated the task description. (Show Details)Feb 6 2023, 2:09 PM

If we're "just" depooling codfw it's worth noting we will still need to depool the affected ms-fe* nodes (since mw always tries to write to both DCs).

jbond updated the task description. (Show Details)Feb 6 2023, 5:21 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-06T22:42:20Z] <inflatador> bking@cumin2002 banning Elastic nodes from cluster in preparation for T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-06T22:48:27Z] <ryankemper> T327925 Banned elastic[2037-2040,2055-2056,2061-2062,2069,2073-2076] on codfw elastic

Icinga downtime and Alertmanager silence (ID=e0e96453-af13-467f-a75e-ebd1c4122a32) set by bking@cumin2002 for 1 day, 0:00:00 on 13 host(s) and their services with reason: switch upgrade

elastic[2037-2040,2055-2056,2061-2062,2069,2073-2076].codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-06T22:55:13Z] <ryankemper> T327925 Depooled codfw wdqs hosts: ryankemper@cumin2002:~$ sudo -E cumin -b 3 'wdqs[2003-2004,2009]*' 'sudo depool'

Staging the new version on the switches:
asw-a-codfw> request system software add force-host set [ /var/tmp/jinstall-ex-4300-21.4R3-S1.5-signed.tgz /var/tmp/jinstall-host-qfx-5-21.4R3-S1.5-signed.tgz ]

This went fine.
Ready for a request system reboot all-members

Change 887284 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/dns@master] admin_state: depool codfw

https://gerrit.wikimedia.org/r/887284

jbond updated the task description. (Show Details)Feb 7 2023, 11:31 AM

herron updated the task description. (Show Details)Feb 7 2023, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-07T12:24:43Z] <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 8:00:00 on doh2001.wikimedia.org with reason: depooled; T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T12:24:58Z] <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on doh2001.wikimedia.org with reason: depooled; T327925

ssingh updated the task description. (Show Details)Feb 7 2023, 12:26 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-07T12:28:55Z] <vgutierrez> depooling authdns2001 - T327925

Vgutierrez updated the task description. (Show Details)Feb 7 2023, 12:38 PM

To depool all services in codfw we will just need to run:

sudo cookbook sre.discovery.datacenter-route --reason 'T327925' depool codfw

from one of the cumin hosts.

Please note: this won't depool docker-registry, which will still be active in codfw for the duration of the maintenance.

Mentioned in SAL (#wikimedia-operations) [2023-02-07T13:05:49Z] <jbond> diable puppet in codfw, ulsfo and esams for switch upgrade T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T13:11:17Z] <oblivian@cumin2002> START - Cookbook sre.discovery.datacenter-route depool all active/active services in codfw: T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T13:12:59Z] <jbond> enable puppet in codfw, ulsfo and esams to allow depools post switch upgrade T327925

jcrespo updated the task description. (Show Details)Feb 7 2023, 1:26 PM

Change 887284 merged by Vgutierrez:

[operations/dns@master] admin_state: depool codfw

https://gerrit.wikimedia.org/r/887284

Mentioned in SAL (#wikimedia-operations) [2023-02-07T13:31:52Z] <vgutierrez> depool codfw edge site - T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T13:32:43Z] <oblivian@cumin2002> END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) depool all active/active services in codfw: T327925

For the record, full row hosts downtime done with:
sudo cookbook sre.hosts.downtime --hours 2 -r "codfw row A upgrade" -t T327925 'P{P:netbox::host%location ~ "A.*codfw"}'

Icinga downtime and Alertmanager silence (ID=295bf4d5-8856-488b-9ca9-06a0ff06db18) set by ayounsi@cumin1001 for 2:00:00 on 199 host(s) and their services with reason: codfw row A upgrade

acmechief2001.codfw.wmnet,acmechief-test2001.codfw.wmnet,aqs[2001-2004].codfw.wmnet,authdns2001.wikimedia.org,backup[2002,2004].codfw.wmnet,cloudbackup2001.codfw.wmnet,contint2001.wikimedia.org,cp[2027-2030].codfw.wmnet,db[2094,2097,2103-2106,2121-2122,2132-2133,2136,2142,2145-2146,2153-2158,2175-2176,2183].codfw.wmnet,dbprov2001.codfw.wmnet,dbproxy2001.codfw.wmnet,doc2001.codfw.wmnet,doh2001.wikimedia.org,elastic[2037-2040,2055-2056,2061-2062,2069,2073-2076].codfw.wmnet,es[2020,2024,2026-2028].codfw.wmnet,ganeti[2023-2024,2027-2030].codfw.wmnet,ganeti-test[2001-2003].codfw.wmnet,gitlab2002.wikimedia.org,grafana2001.codfw.wmnet,kafka-logging2001.codfw.wmnet,kafka-main2001.codfw.wmnet,kafkamon2002.codfw.wmnet,kubemaster2001.codfw.wmnet,kubernetes[2005,2007-2008,2018-2019].codfw.wmnet,kubestage2001.codfw.wmnet,kubetcd2004.codfw.wmnet,logstash[2001,2023,2026,2033].codfw.wmnet,lvs2007.codfw.wmnet,maps2005.codfw.wmnet,mc[2038-2041,2055].codfw.wmnet,mc-gp2001.codfw.wmnet,ml-cache2001.codfw.wmnet,ml-serve[2001,2005].codfw.wmnet,ml-staging2001.codfw.wmnet,ml-staging-etcd2001.codfw.wmnet,moss-be2001.codfw.wmnet,ms-be[2040,2044-2045,2051-2052,2060,2062,2066].codfw.wmnet,ms-fe2009.codfw.wmnet,mw[2291-2309,2377-2411].codfw.wmnet,mwdebug2001.codfw.wmnet,ncredir2001.codfw.wmnet,netbox2002.codfw.wmnet,netboxdb2002.codfw.wmnet,netmon2002.wikimedia.org,ores[2001-2002].codfw.wmnet,orespoolcounter2003.codfw.wmnet,parse[2001-2005].codfw.wmnet,pc2011.codfw.wmnet,people2002.codfw.wmnet,pki2001.codfw.wmnet,planet2002.codfw.wmnet,poolcounter2003.codfw.wmnet,puppetdb2002.codfw.wmnet,puppetmaster[2001,2004].codfw.wmnet,rdb2007.codfw.wmnet,registry2003.codfw.wmnet,rpki2002.codfw.wmnet,test-reimage2001.codfw.wmnet,testvm[2002,2004-2005].codfw.wmnet,thanos-be2001.codfw.wmnet,thanos-fe2001.codfw.wmnet,thumbor2005.codfw.wmnet,urldownloader2001.wikimedia.org,wdqs[2003-2004,2009].codfw.wmnet,xhgui2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-07T13:56:16Z] <Emperor> depool ms-fe2009 T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T13:59:57Z] <XioNoX> disable puppet in ulsfo/esams/codfw for codfw row A switch upgrade - T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T14:06:51Z] <XioNoX> asw-a-codfw> request system reboot all-members - T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T14:08:45Z] <jbond> disable puppet in codfw, uslfo, esams for switch upgrade T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T14:26:37Z] <claime> depooled appserver, api_appserver, jobrunner, parsoid - T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T14:27:59Z] <jbond> enable puppet in codfw, uslfo, esams post switch upgrade T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T14:32:20Z] <Emperor> pool ms-fe2009 (codfw as a whole still depooled) T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T14:36:25Z] <claime> repooled appserver, api_appserver, jobrunner, parsoid - T327925

Change 886984 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/dns@master] Revert "admin_state: depool codfw"

https://gerrit.wikimedia.org/r/886984

Mentioned in SAL (#wikimedia-operations) [2023-02-07T14:46:49Z] <volans@cumin2002> START - Cookbook sre.discovery.datacenter-route pool all active/active services in codfw: T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T15:00:53Z] <vgutierrez> restart pybal in lvs2009 - T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T15:04:37Z] <vgutierrez> restart pybal in lvs2010 - T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T15:07:56Z] <volans@cumin2002> END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) pool all active/active services in codfw: T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T15:09:02Z] <volans@cumin2002> START - Cookbook sre.discovery.service-route depool restbase-async in eqiad: T327925

Change 886984 merged by Vgutierrez:

[operations/dns@master] Revert "admin_state: depool codfw"

https://gerrit.wikimedia.org/r/886984

Mentioned in SAL (#wikimedia-operations) [2023-02-07T15:12:21Z] <vgutierrez> repool codfw edge site - T327925

Mentioned in SAL (#wikimedia-operations) [2023-02-07T15:14:06Z] <volans@cumin2002> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool restbase-async in eqiad: T327925

Clement_Goubert updated the task description. (Show Details)Feb 7 2023, 3:28 PM

The upgrade was smooth, ~15min hard downtime.
No user impact, all the depools did their job. There was some paging alerts that got discussed on IRC, not sure what the conclusion was.

Thanks everybody!

colewhite updated the task description. (Show Details)Feb 7 2023, 3:49 PM

Change 886812 merged by Jcrespo:

[operations/puppet@production] Revert "dbbackups: Delay codfw es (db content) backups by one day"

https://gerrit.wikimedia.org/r/886812

Maintenance_bot moved this task from In progress to Done on the DBA board.Feb 7 2023, 4:15 PM

Maintenance_bot removed a project: Patch-For-Review.Feb 7 2023, 4:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-07T17:55:48Z] <inflatador> bking@cumin1001 repooling elastic and wdqs hosts post-maintenance T327925

calbon moved this task from Watching to 2023-2024 Q3 Done on the Machine-Learning-Team board.Nov 29 2023, 2:20 PM

codfw row A switches upgradeClosed, ResolvedPublicActions

Description

codfw row A switches upgrade