| jcrespo | |
| Mar 4 2025, 4:46 PM |
| F62387685: image.png | |
| Jun 19 2025, 7:37 AM |
| F58618382: image.png | |
| Mar 6 2025, 10:16 AM |
Rather than upgrading these hosts, we will decommission by setting up different hosts and upgrade those (or set them up already upgraded for the first time):
Pending actions:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jcrespo | T376916 Upgrade backup hosts to Debian Bookworm 12.X | |||
| Resolved | jcrespo | T387892 Decommission backup1001, backup1002, backup2001, backup2002 (and their arrays) | |||
| Resolved | Request | Jclark-ctr | T398185 decommission backup1001 and its 2 disk arrays | ||
| Resolved | Request | Jhancock.wm | T398188 decommission backup2001 and its 2 disk arrays | ||
| Resolved | Request | Jclark-ctr | T398210 decommission backup1002 and its disk array | ||
| Resolved | Request | Jhancock.wm | T398212 decommission backup2002 and its disk array |
Change #1124487 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] dbbackups: Prepare backup1013 to take over eqiad backups of es* dbs
Change #1124487 merged by Jcrespo:
[operations/puppet@production] dbbackups: Prepare backup1013 to take over eqiad backups of es* dbs
Change #1124720 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] dbbackups: Prepare backup2013 to take over codfw backups of es* dbs
Change #1124720 merged by Jcrespo:
[operations/puppet@production] dbbackups: Prepare backup2013 to take over codfw backups of es* dbs
Mentioned in SAL (#wikimedia-operations) [2025-03-05T09:18:53Z] <jynus> deploy new backup grants for es1036,es1040 T387892
Mentioned in SAL (#wikimedia-operations) [2025-03-05T09:23:07Z] <jynus> deploy new backup grants for es2036,es2040 T387892
Change #1124738 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] dbbackups: Migrate es backups from backup[12]02 to backup[12]13
Change #1124738 merged by Jcrespo:
[operations/puppet@production] dbbackups: Migrate es backups from backup[12]02 to backup[12]13
Mentioned in SAL (#wikimedia-operations) [2025-03-05T15:40:35Z] <jynus> starting es backups on new hosts backup1013, backup2013 T387892
I made a mistake, and it was that I didn't add the new hosts into the production grants of m1 (for backup state tracking). No fatal issue (metadata gathering is optional and won't be a hard failure), but it will mean we will have to fix that tomorrow and test it again.
Change #1124834 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] dbbackups: Add additional m1 grants for backup[12]013 stats user
Mentioned in SAL (#wikimedia-operations) [2025-03-06T09:28:35Z] <jynus> deploy additional grants to m1 T387892
Change #1124834 merged by Jcrespo:
[operations/puppet@production] dbbackups: Add additional m1 grants for backup[12]013 stats user
Change #1125114 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] dbbackups: Prepare backup1002, backup2002 for decommissioning
Mentioned in SAL (#wikimedia-operations) [2025-03-12T10:14:28Z] <jynus> removing backup1002, backup2002 dump user on es6,es7 T387892
Mentioned in SAL (#wikimedia-operations) [2025-03-12T10:42:21Z] <jynus> removing backup1002, backup2002 dbbackups user @ m1 T387892
Change #1125114 merged by Jcrespo:
[operations/puppet@production] dbbackups: Prepare backup1002, backup2002 for decommissioning
Change #1160691 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Migrate backup1001's director role to backup1014
Mentioned in SAL (#wikimedia-operations) [2025-06-18T10:14:49Z] <jynus> starting backup director migration backup1001 -> backup1014 T387892
Change #1160691 merged by Jcrespo:
[operations/puppet@production] bacula: Migrate backup1001's director role to backup1014
Icinga downtime and Alertmanager silence (ID=6ac28898-4cc3-4a9a-b6a4-de2ec38f7f20) set by jynus@cumin1002 for 4:00:00 on 2 host(s) and their services with reason: Backup director migration
backup[1001,1014].eqiad.wmnet
Change #1160730 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Update wrong role for backup2009
Change #1160730 merged by Jcrespo:
[operations/puppet@production] bacula: Update wrong role for backup2009
Change #1160735 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Force puppet7 on backup1009
Change #1160735 merged by Jcrespo:
[operations/puppet@production] bacula: Force puppet7 on backup1009
Change #1160739 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Migrate and create general properties for the new/renamed roles
Change #1160739 merged by Jcrespo:
[operations/puppet@production] bacula: Migrate and create general properties for the new/renamed roles
Change #1160756 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Discourage the usage of backup1001 as director
Change #1160756 merged by Jcrespo:
[operations/puppet@production] bacula: Discourage the usage of backup1001 as director
Bacula director migrated, with some bumps along the way.
Documentation updated.
Issue updated to say:
Debian GNU/Linux 11 (bullseye)
_ _ _ ___ _____ _ _ _
__| | ___ | \ | |/ _ \_ _| _ _ ___ ___ | |_| |__ (_)___
/ _` |/ _ \ | \| | | | || | | | | / __|/ _ \ | __| '_ \| / __|
| (_| | (_) | | |\ | |_| || | | |_| \__ \ __/ | |_| | | | \__ \
\__,_|\___/ |_| \_|\___/ |_| \__,_|___/\___| \__|_| |_|_|___/
_
___ ___ _ ____ _____ _ __| |
/ __|/ _ \ '__\ \ / / _ \ '__| |
\__ \ __/ | \ V / __/ | |_|
|___/\___|_| \_/ \___|_| (_)
backup1001 is no longer the active bacula director for WMF fails.
Use backup1014 instead !!!Mentioned in SAL (#wikimedia-operations) [2025-06-18T13:00:40Z] <jynus> bacula director migration finalized, backup1014 is the new bacula director. backup1001 should no longer be used. T387892
backup1001 and backup2001 are also closer to decom, I need first to review the storage contents to make sure I don't miss anything not migrated (at least the archive storage is pending), and we will be able to run the script.
regarding *002 hosts, I am still waiting on dbas to confirm the refresh of es read only backups.
Mentioned in SAL (#wikimedia-operations) [2025-06-19T07:37:45Z] <jynus> just started es read only backup regeneration T387892
Mentioned in SAL (#wikimedia-operations) [2025-06-23T15:47:13Z] <jynus> drop backup users from es1-es5 hosts T387892
Change #1163645 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] dbbackups: Enable temporarily read only backups for refresh
Change #1163645 merged by Jcrespo:
[operations/puppet@production] dbbackups: Enable temporarily read only backups for refresh
Change #1163694 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] dbbackups: Disable read only backups and reenable regular rw es backups
Change #1164158 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Create a temporary backup job for long term Archival
Change #1164158 merged by Jcrespo:
[operations/puppet@production] bacula: Create a temporary backup job for long term Archival
Change #1163694 merged by Jcrespo:
[operations/puppet@production] dbbackups: Disable read only backups and reenable regular rw es backups
backups for es ro are almost complete (except es5 for backup2013), almost ready to decom backup[12]002.
Migration of archival files from backup1001 also ongoing and should finish today.
This should unblock decommissioning of all hosts for next week.
Change #1164981 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Remove oldmain and olddirector roles, prepare for decom backup[12]01
Change #1164981 merged by Jcrespo:
[operations/puppet@production] bacula: Remove oldmain and olddirector roles, prepare for decom backup[12]01
Change #1165822 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Remove backup1001 old backup director host from puppet
Change #1165823 had a related patch set uploaded (by Jcrespo; author: Jcrespo):
[operations/puppet@production] bacula: Remove backup2001, old offsite backup host
Change #1165823 merged by Jcrespo:
[operations/puppet@production] bacula: Remove backup2001, old offsite backup host
Change #1165822 merged by Jcrespo:
[operations/puppet@production] bacula: Remove backup1001 old backup director host from puppet
This is now finished, with just some operations to be done by dc-ops on the children tasks. From a backup perspective, the migration is complete.
