Page MenuHomePhabricator
Create Task
Maniphest T298585

Upgrade WMF database-and-backup-related hosts to bullseye
Closed, ResolvedPublic
Actions

Description

Tracking the progress our progress of upgrading all databases, and services around them (monitoring, backups, etc.) to Debian Bulsseye and MariaDB 10.4.

This task is stalled until we are completely happy with the testing phase: T295965

Details

Other Assignee
LSobanski

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT291916 Tracking task for Bullseye migrations in production
 Resolved• MarosteguiT298585 Upgrade WMF database-and-backup-related hosts to bullseye
 Resolved• MarosteguiT295965 Test MariaDB 10.4 with Bullseye
 Resolved• CmjohnsonT299025 db1169 reimage/idrac failure
 Resolved• CmjohnsonT299123 es1022 troubles with PXE
 Resolved• MarosteguiT298586 Upgrade all dbproxy hosts to Bullseye
 ResolvedBTullisT298940 Reimage WMCS db proxies to Bullseye
 Resolved• MarosteguiT298769 Upgrade x2 to Bullseye
 ResolvedjhathawayT298959 Upgrade dborch1001 to Bullseye
 Resolved• MarosteguiT299046 Upgrade parsercache infra to Bullseye
 Resolved• MarosteguiT299344 Upgrade m1 to Bullseye
 Resolved• MarosteguiT299624 Switchover m1 master (db1159 -> db1128)
 Resolved• MarosteguiT299479 Upgrade s6 to Bullseye
 Resolved• MarosteguiT300471 Switchover s6 master (db1173 -> db1131)
 DeclinedTrizek-WMFT302283 Read-only window needed for s6
 Resolved• CmjohnsonT310595 db1173 won't boot up
Resolved• razziT299480 Upgrade clouddb* hosts to Bullseye
 Resolved• razziT299481 Upgrade dbstore100* hosts to Bullseye
 Resolved• MarosteguiT299741 Upgrade es1 to Bullseye
 ResolvedjcrespoT299876 Upgrade database backup sources and dbprov* hosts to Bullseye + MariaDB 10.4
 Resolved• MarosteguiT299889 Upgrade es2 to Bullseye
 ResolvedLadsgroupT299908 Change "Worst response time" graph after migrating to Bullseye
 ResolvedLadsgroupT299911 Upgrade es3 to Bullseye
 Resolved• MarosteguiT300005 Upgrade es4 to Bullseye
 Resolved• MarosteguiT300127 Switchover es4 master es1021 -> es1020
 ResolvedLadsgroupT300006 Upgrade es5 to Bullseye
 ResolvedLadsgroupT300976 Switchover es5 master
 Resolved• MarosteguiT300099 Upgrade x1 to Bullseye
 Resolved• MarosteguiT300472 Switchover x1 master (db1103 -> db1120)
 Resolved• MarosteguiT300243 Upgrade m2 to Bullseye
 Resolved• MarosteguiT300329 Switchover m2 master (db1183 -> db1159)
 ResolvedLadsgroupT300510 Upgrade s2 to Bullseye
 Resolved• MarosteguiT306417 Switchover s2 master (db1122 -> db1162)
 Resolved• MarosteguiT300600 Upgrade s3 to Bullseye
 Resolved• MarosteguiT301850 Switchover s3 master (db1157 -> db1123)
 Resolved• CmjohnsonT304280 db1175 not booting up
 Resolved• MarosteguiT300473 Upgrade s5 to Bullseye
 Resolved• MarosteguiT303798 Switchover s5 master db1130 -> db1100
 Resolved• MarosteguiT300835 Upgrade m3 to Bullseye
 Resolved• MarosteguiT301219 Switchover m3 master (db1107 -> db1183)
 Duplicate• KormatT301653 Upgrade s7 to Bullseye
 Resolved• MarosteguiT301654 Upgrade m5 to Bullseye
 Resolved• MarosteguiT302190 Switchover m5 master (db1132 -> db1107)
 ResolvedLadsgroupT302185 Upgrade s8 to Bullseye
 ResolvedLadsgroupT303927 Switchover s8 master (db1109 -> db1104)
 ResolvedLadsgroupT302363 Upgrade s7 to bullseye
 Resolved• MarosteguiT306001 Switchover s7 master (db1136 -> db1181)
 ResolvedLadsgroupT302950 Upgrade s4 to bullseye
 Resolved• MarosteguiT304933 Switchover s4 master (db1138 -> db1160)
 ResolvedLadsgroupT303171 Upgrade s1 to Bullseye
 Resolved• CmjohnsonT307198 db1164 fails to POST/boot/etc
 ResolvedLadsgroupT301312 Switchover s1 master (db1118 -> db1163)
 ResolvedJclark-ctrT308246 db1164 power supply isn't redundant
DeclinedBTullisT304492 Upgrade db1108 to Bullseye
 ResolvedjcrespoT305446 Upgrade backup* hosts to bullseye

Event Timeline

• Marostegui created this task.Jan 5 2022, 5:43 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2022, 5:43 AM
• Marostegui changed the task status from Open to Stalled.Jan 5 2022, 5:43 AM
• Marostegui triaged this task as Medium priority.
• Marostegui moved this task from Triage to Blocked on the DBA board.
• Marostegui added a subtask: T295965: Test MariaDB 10.4 with Bullseye.
• Kormat subscribed.Jan 10 2022, 11:09 AM
• Marostegui changed the status of subtask T299046: Upgrade parsercache infra to Bullseye from Open to Stalled.Jan 12 2022, 12:39 PM
• Marostegui changed the status of subtask T299046: Upgrade parsercache infra to Bullseye from Stalled to Open.Jan 14 2022, 6:59 AM
• Marostegui closed subtask T298586: Upgrade all dbproxy hosts to Bullseye as Resolved.Jan 17 2022, 9:23 AM
• Marostegui closed subtask T298769: Upgrade x2 to Bullseye as Resolved.Jan 17 2022, 12:54 PM
• Marostegui changed the task status from Stalled to Open.Jan 19 2022, 6:05 AM
• Marostegui moved this task from Blocked to Meta/Epic on the DBA board.

This can now proceed, all the tests were fine

• Marostegui closed subtask T295965: Test MariaDB 10.4 with Bullseye as Resolved.Jan 19 2022, 6:05 AM
• Marostegui closed subtask T299741: Upgrade es1 to Bullseye as Resolved.Jan 24 2022, 10:28 AM
Ladsgroup closed subtask T299908: Change "Worst response time" graph after migrating to Bullseye as Resolved.Jan 24 2022, 2:29 PM
• Marostegui closed subtask T299046: Upgrade parsercache infra to Bullseye as Resolved.Jan 25 2022, 8:38 AM
• Marostegui closed subtask T299889: Upgrade es2 to Bullseye as Resolved.Jan 25 2022, 2:21 PM
Ladsgroup closed subtask T299911: Upgrade es3 to Bullseye as Resolved.Jan 25 2022, 7:13 PM
• Marostegui closed subtask T299344: Upgrade m1 to Bullseye as Resolved.Jan 27 2022, 10:33 AM
• Marostegui added a subtask: T300473: Upgrade s5 to Bullseye.Feb 1 2022, 10:19 AM
• Marostegui closed subtask T300005: Upgrade es4 to Bullseye as Resolved.Feb 2 2022, 10:36 AM
• Marostegui closed subtask T300243: Upgrade m2 to Bullseye as Resolved.Feb 3 2022, 9:31 AM
• Marostegui added a subtask: T300835: Upgrade m3 to Bullseye.Feb 3 2022, 11:11 AM
• Marostegui updated Other Assignee, added: LSobanski.Feb 3 2022, 1:30 PM
• Marostegui closed subtask T300835: Upgrade m3 to Bullseye as Resolved.Feb 15 2022, 8:05 AM
Ladsgroup closed subtask T300006: Upgrade es5 to Bullseye as Resolved.Feb 15 2022, 6:20 PM
Ladsgroup changed the status of subtask T300510: Upgrade s2 to Bullseye from Open to Stalled.Feb 17 2022, 7:59 PM
Jdforrester-WMF added a parent task: T291916: Tracking task for Bullseye migrations in production.Feb 23 2022, 4:49 PM
Jdforrester-WMF mentioned this in T247045: Migrate all of production metal and VMs to Buster or later.Feb 23 2022, 4:53 PM
Jdforrester-WMF mentioned this in T291916: Tracking task for Bullseye migrations in production.
• Marostegui closed subtask T301654: Upgrade m5 to Bullseye as Resolved.Mar 9 2022, 2:11 PM
• Marostegui closed subtask T300600: Upgrade s3 to Bullseye as Resolved.Mar 29 2022, 9:47 AM
• Marostegui closed subtask T300473: Upgrade s5 to Bullseye as Resolved.Apr 4 2022, 5:34 AM
• Marostegui closed subtask T299481: Upgrade dbstore100* hosts to Bullseye as Resolved.Apr 6 2022, 4:42 AM
jcrespo closed subtask T299876: Upgrade database backup sources and dbprov* hosts to Bullseye + MariaDB 10.4 as Resolved.Apr 8 2022, 12:59 PM
jcrespo subscribed.

dbprov, backup sources, backup test hosts and mediabackupstemp upgraded to bullseye (T299876).

• Marostegui closed subtask T302950: Upgrade s4 to bullseye as Resolved.Apr 12 2022, 7:02 AM
• Marostegui closed subtask T302363: Upgrade s7 to bullseye as Resolved.Apr 20 2022, 4:35 AM
Ladsgroup closed subtask T302185: Upgrade s8 to Bullseye as Resolved.Apr 26 2022, 6:47 AM
Ladsgroup changed the status of subtask T300510: Upgrade s2 to Bullseye from Stalled to Open.Apr 26 2022, 6:50 AM
• Marostegui closed subtask T300510: Upgrade s2 to Bullseye as Resolved.Apr 26 2022, 8:26 AM
• razzi closed subtask T299480: Upgrade clouddb* hosts to Bullseye as Resolved.Apr 26 2022, 5:21 PM
• Kormat changed the status of subtask T303171: Upgrade s1 to Bullseye from Open to Stalled.May 2 2022, 7:24 AM
jcrespo closed subtask T305446: Upgrade backup* hosts to bullseye as Resolved.May 12 2022, 11:13 AM
• Marostegui changed the status of subtask T303171: Upgrade s1 to Bullseye from Stalled to Open.May 12 2022, 1:33 PM
Ladsgroup closed subtask T303171: Upgrade s1 to Bullseye as Resolved.May 23 2022, 6:25 PM
• Marostegui closed subtask T299479: Upgrade s6 to Bullseye as Resolved.Jun 15 2022, 5:21 AM
• Marostegui closed subtask T300099: Upgrade x1 to Bullseye as Resolved.Jun 30 2022, 7:48 AM
• Marostegui closed this task as Resolved.Jun 30 2022, 7:55 AM
• Marostegui claimed this task.

This is all done.
Pending only hosts we do not own and have their own tracking task:

db1108 T304492
dbproxy1018 - dbproxy1019 T298940

Closing this.
Thanks everyone for all the work in getting this done!

jhathaway closed subtask T298959: Upgrade dborch1001 to Bullseye as Resolved.Mar 22 2023, 7:23 PM
• Marostegui closed subtask T304492: Upgrade db1108 to Bullseye as Declined.Apr 5 2023, 8:21 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits