Page MenuHomePhabricator
Create Task
Maniphest T389380

Upgrade Cumin hosts to Bookworm
Closed, ResolvedPublic
Actions

Description

We can create a parallel cumin1003 VM on Bookworm, test/adapt everything, then reimage cumin2002 and finally decom cumin1002.

  • Create cumin1003
  • Upgrade cuminunpriv1001
  • Upgrade cumin2002
  • Migrate pwstore repository to cumin1003
  • Migrate backups to cumin1003
  • Copy old cookbook/spicerack logs
  • Decom cumin1002

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Enable nftables on cluster::management on the role leveloperations/puppetproduction+1 -2
Remove cumin1002 from list of Cumin mastersoperations/puppetproduction+0 -2
wmf_root_client.pp: Remove cumin1002operations/puppetproduction+1 -1
Setup cumin1002 to insetupoperations/puppetproduction+1 -1
Remove cumin1002 as Homer git peeroperations/puppetproduction+0 -2
Remove cumin1002 from mysql root listoperations/puppetproduction+0 -1
Update pwstore docs to point to cumin1003operations/debs/wmf-laptopmaster+1 -1
Remove cumin1002 from alertmanager accessoperations/puppetproduction+0 -1
Remove cumin1002 from tcpircbot configoperations/puppetproduction+0 -3
production-ms.sql.erb: Remove root@10.64.48.98operations/puppetproduction+0 -2
Remove grant from cumin1002operations/puppetproduction+0 -7
Ganeti: Remove cumin1002 from allow list for RAPI accessoperations/puppetproduction+1 -4
Add safe.directory directives for the pwstore repositoryoperations/puppetproduction+10 -0
Also switch cumin2002 to nftablesoperations/puppetproduction+1 -0
transferpy: Build for Bookwormoperations/software/transferpymaster+6 -0
homer-diff-checker: move execution from cumin1002 to cumin1003operations/puppetproduction+2 -2
cumin: Migrate cumin1002 mariadb remote backups to cumin1003operations/puppetproduction+3 -1
dbbackups: Advance snapshot dbbackups start time by 4 hoursoperations/puppetproduction+1 -1
homer: fix private repository configoperations/puppetproduction+4 -0
homer: make private repo support multiple peersoperations/puppetproduction+31 -26
Make cumin1003 a Cumin nodeoperations/puppetproduction+16 -2
dnsdisc: make it compatible with bookwormoperations/software/spicerackmaster+17 -11
Add cumin1003 to site.ppoperations/puppetproduction+6 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.Mar 31 2025, 6:30 AM
ops-monitoring-bot added a comment.Mar 31 2025, 8:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host cumin1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 31 2025, 8:47 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host cumin1003.eqiad.wmnet with OS bookworm completed:

  • cumin1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503310832_jmm_380057_cumin1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 31 2025, 10:25 AM

Change #1132577 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make cumin1003 a Cumin node

https://gerrit.wikimedia.org/r/1132577

gerritbot added a project: Patch-For-Review.Mar 31 2025, 10:25 AM
gerritbot added a comment.Apr 3 2025, 8:24 AM

Change #1133808 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/spicerack@master] dnsdisc: make it compatible with bookworm

https://gerrit.wikimedia.org/r/1133808

gerritbot added a comment.Apr 14 2025, 3:45 PM

Change #1133808 merged by jenkins-bot:

[operations/software/spicerack@master] dnsdisc: make it compatible with bookworm

https://gerrit.wikimedia.org/r/1133808

elukey closed subtask T390862: Kafka dependency upgrade in spicerack as Resolved.Apr 29 2025, 1:21 PM
elukey closed subtask T390864: Redis dependency upgrade in spicerack as Resolved.Apr 29 2025, 1:39 PM
elukey closed subtask T390857: Kubernetes dependency upgrade in spicerack as Resolved.May 5 2025, 2:04 PM
Stashbot added a comment.May 7 2025, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-07T15:53:40Z] <moritzm> uploaded a python-pynetbox 7.4.1-1~wmf12u1 to bookworm-wikimedia (needed for Cumin update) T389380

gerritbot added a comment.May 8 2025, 11:41 AM

Change #1143539 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/software/transferpy@master] transferpy: Build for Bookworm

https://gerrit.wikimedia.org/r/1143539

Stashbot added a comment.May 8 2025, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T11:57:51Z] <moritzm> import transferpy 1.1+deb12u1 to bookworm-wikimedia T389380

gerritbot added a comment.May 8 2025, 1:16 PM

Change #1132577 merged by Muehlenhoff:

[operations/puppet@production] Make cumin1003 a Cumin node

https://gerrit.wikimedia.org/r/1132577

Stashbot added a comment.May 8 2025, 2:45 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T14:45:06Z] <moritzm> imported ripe-atlas-sagan 1.3.1-1~wmf12u1 to apt.wikimedia.org/bookworm T389380

Stashbot added a comment.May 8 2025, 2:45 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T14:45:27Z] <moritzm> imported ripe-atlas-tools 2.3.0-3+wmf12u1 to apt.wikimedia.org/bookworm T389380

MoritzMuehlenhoff added a comment.May 8 2025, 2:47 PM

ripe-atlas-tools was in Bullseye, but missed Bookworm due to some RC bug. It's again part of Debian with trixie, so I prepared an internal build of 2.3.0-3+wmf12u1 for Bookworm and uploaded it to apt.wikimedia.org. This also needed a backport of
ripe-atlas-sagan (1.3.1-1~wmf12u1)

ops-monitoring-bot added a comment.May 9 2025, 5:30 AM

Icinga downtime and Alertmanager silence (ID=4e0ba7e4-66f2-4cc7-8562-eed30c2476f1) set by jmm@cumin2002 for 5 days, 0:00:00 on 1 host(s) and their services with reason: WIP new Bookworm host

cumin1003.eqiad.wmnet
Stashbot mentioned this in T393711: httpbb bookworm support.May 16 2025, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-16T08:26:02Z] <moritzm> uploaded httpbb 0.0.5-1+deb12u1 to apt.wikimedia.org T393711 T389380

gerritbot added a comment.May 20 2025, 8:15 AM

Change #1148268 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] homer: make private repo support multiple peers

https://gerrit.wikimedia.org/r/1148268

gerritbot added a comment.May 26 2025, 12:50 PM

Change #1148268 merged by Volans:

[operations/puppet@production] homer: make private repo support multiple peers

https://gerrit.wikimedia.org/r/1148268

gerritbot added a comment.May 26 2025, 1:04 PM

Change #1150675 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] homer: fix private repository config

https://gerrit.wikimedia.org/r/1150675

gerritbot added a comment.May 26 2025, 1:09 PM

Change #1150675 merged by Volans:

[operations/puppet@production] homer: fix private repository config

https://gerrit.wikimedia.org/r/1150675

FCeratto-WMF changed the status of subtask T393990: Deploy grants for cumin1003 from Open to In Progress.Jun 4 2025, 10:35 AM
gerritbot added a comment.Jun 16 2025, 12:39 PM

Change #1159439 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Advance snapshot dbbackups start time by 4 hours

https://gerrit.wikimedia.org/r/1159439

gerritbot added a comment.Jun 16 2025, 2:01 PM

Change #1159439 merged by Jcrespo:

[operations/puppet@production] dbbackups: Advance snapshot dbbackups start time by 4 hours

https://gerrit.wikimedia.org/r/1159439

MoritzMuehlenhoff updated the task description. (Show Details)Jun 17 2025, 11:51 AM
gerritbot added a comment.Jun 17 2025, 1:11 PM

Change #1160133 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Also switch cumin2002 to nftables

https://gerrit.wikimedia.org/r/1160133

MoritzMuehlenhoff updated the task description. (Show Details)Jun 18 2025, 7:12 AM
MoritzMuehlenhoff added a subtask: T393692: transfer.py fails when handling nftables-configured firewall.
MoritzMuehlenhoff updated the task description. (Show Details)Jun 18 2025, 2:26 PM
Volans added a comment.Jun 18 2025, 8:45 PM

I had to deploy homer to cumin2002 after the upgrade:

sudo cookbook sre.deploy.python-code -r 'Release v0.10.1' homer 'cumin2002*'

Now it works fine.

FCeratto-WMF mentioned this in T393990: Deploy grants for cumin1003.Jun 25 2025, 12:31 PM
MoritzMuehlenhoff closed subtask T393711: httpbb bookworm support as Resolved.Jul 10 2025, 8:31 AM
FCeratto-WMF closed subtask T393990: Deploy grants for cumin1003 as Resolved.Jul 21 2025, 9:11 AM
Volans mentioned this in T388874: Update Kubernetes library version in spicerack.Jul 21 2025, 3:54 PM
MoritzMuehlenhoff updated the task description. (Show Details)Aug 28 2025, 12:23 PM
bking subscribed.Sep 10 2025, 9:05 PM
jcrespo updated the task description. (Show Details)Oct 17 2025, 11:07 AM
gerritbot added a comment.Oct 17 2025, 11:08 AM

Change #1196886 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] cumin: Migrate cumin1002 mariadb remote backups to cumin1003

https://gerrit.wikimedia.org/r/1196886

jcrespo subscribed.Oct 17 2025, 12:41 PM

I removed the nftables on transfer.py checklist- it is important (and I am working on that now), but it only limits the hosts it can transfer to, not where it can be installed (not a hard dependency).

gerritbot added a comment.Oct 20 2025, 8:38 AM

Change #1196886 merged by Jcrespo:

[operations/puppet@production] cumin: Migrate cumin1002 mariadb remote backups to cumin1003

https://gerrit.wikimedia.org/r/1196886

jcrespo updated the task description. (Show Details)Oct 20 2025, 10:01 AM

The backup migration was deployed and tested succesfuly. But please keep cumin1002 for a few extra days while I keep monitoring it, in case there is some more subtle breakage monitoring didn't catch.

gerritbot added a comment.Oct 20 2025, 6:23 PM

Change #1197321 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] homer-diff-checker: move execution from cumin1002 to cumin1003

https://gerrit.wikimedia.org/r/1197321

gerritbot added a comment.Oct 21 2025, 7:56 AM

Change #1197321 merged by Cathal Mooney:

[operations/puppet@production] homer-diff-checker: move execution from cumin1002 to cumin1003

https://gerrit.wikimedia.org/r/1197321

gerritbot added a comment.Nov 3 2025, 11:26 AM

Change #1143539 abandoned by Jcrespo:

[operations/software/transferpy@master] transferpy: Build for Bookworm

Reason:

Integrated elsewhere

https://gerrit.wikimedia.org/r/1143539

jcrespo closed subtask T393692: transfer.py fails when handling nftables-configured firewall as Resolved.Nov 5 2025, 2:19 PM

No blockers from me to remove cumin1002 (db backup orchestration was migrated already).

elukey closed subtask T390860: Elasticsearch dependency upgrade in spicerack as Resolved.Nov 6 2025, 9:32 AM
elukey subscribed.

New spicerack release deployed, cumin1002 is not needed anymore from Data Platform folks.

MoritzMuehlenhoff reopened subtask T390860: Elasticsearch dependency upgrade in spicerack as Open.Nov 10 2025, 5:39 PM
gerritbot added a comment.Nov 12 2025, 10:35 AM

Change #1204357 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add safe.directory directives for the pwstore repository

https://gerrit.wikimedia.org/r/1204357

gerritbot added a comment.Nov 12 2025, 11:12 AM

Change #1160133 merged by Muehlenhoff:

[operations/puppet@production] Also switch cumin2002 to nftables

https://gerrit.wikimedia.org/r/1160133

Stashbot added a comment.Nov 12 2025, 11:36 AM

Mentioned in SAL (#wikimedia-operations) [2025-11-12T11:36:47Z] <moritzm> migrated cumin2002 to nftables T389380

gerritbot added a comment.Nov 12 2025, 11:48 AM

Change #1204368 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable nftables on cluster::management on the role level

https://gerrit.wikimedia.org/r/1204368

gerritbot added a comment.Nov 12 2025, 11:54 AM

Change #1204357 merged by Muehlenhoff:

[operations/puppet@production] Add safe.directory directives for the pwstore repository

https://gerrit.wikimedia.org/r/1204357

gerritbot added a comment.Nov 12 2025, 12:36 PM

Change #1204375 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/debs/wmf-laptop@master] Update pwstore docs to point to cumin1003

https://gerrit.wikimedia.org/r/1204375

MoritzMuehlenhoff updated the task description. (Show Details)Nov 12 2025, 12:37 PM
gerritbot added a comment.Nov 12 2025, 12:46 PM

Change #1204380 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Ganeti: Remove cumin1002 from allow list for RAPI access

https://gerrit.wikimedia.org/r/1204380

gerritbot added a comment.Nov 12 2025, 1:19 PM

Change #1204574 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove grant from cumin1002

https://gerrit.wikimedia.org/r/1204574

gerritbot added a comment.Nov 12 2025, 3:17 PM

Change #1204380 merged by Muehlenhoff:

[operations/puppet@production] Ganeti: Remove cumin1002 from allow list for RAPI access

https://gerrit.wikimedia.org/r/1204380

Stashbot added a comment.Nov 12 2025, 3:17 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-12T15:17:48Z] <moritzm> migrated pwstore repository from cumin1002 to cumin1003 T389380

gerritbot added a comment.Nov 12 2025, 3:31 PM

Change #1204609 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove cumin1002 from tcpircbot config

https://gerrit.wikimedia.org/r/1204609

gerritbot added a comment.Nov 12 2025, 3:42 PM

Change #1204574 merged by Marostegui:

[operations/puppet@production] Remove grant from cumin1002

https://gerrit.wikimedia.org/r/1204574

gerritbot added a comment.Nov 12 2025, 3:45 PM

Change #1204617 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] production-ms.sql.erb: Remove root@10.64.48.98

https://gerrit.wikimedia.org/r/1204617

gerritbot added a comment.Nov 12 2025, 3:47 PM

Change #1204617 merged by Marostegui:

[operations/puppet@production] production-ms.sql.erb: Remove root@10.64.48.98

https://gerrit.wikimedia.org/r/1204617

gerritbot added a comment.Nov 12 2025, 4:08 PM

Change #1204620 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove cumin1002 from alertmanager access

https://gerrit.wikimedia.org/r/1204620

gerritbot added a comment.Nov 12 2025, 4:11 PM

Change #1204622 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove cumin1002 as Homer git peer

https://gerrit.wikimedia.org/r/1204622

gerritbot added a comment.Nov 12 2025, 4:13 PM

Change #1204628 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] wmf_root_client.pp: Remove cumin1002

https://gerrit.wikimedia.org/r/1204628

gerritbot added a comment.Nov 13 2025, 8:06 AM

Change #1204609 merged by Muehlenhoff:

[operations/puppet@production] Remove cumin1002 from tcpircbot config

https://gerrit.wikimedia.org/r/1204609

gerritbot added a comment.Nov 13 2025, 8:20 AM

Change #1204620 merged by Muehlenhoff:

[operations/puppet@production] Remove cumin1002 from alertmanager access

https://gerrit.wikimedia.org/r/1204620

gerritbot added a comment.Nov 13 2025, 8:25 AM

Change #1204375 merged by Muehlenhoff:

[operations/debs/wmf-laptop@master] Update pwstore docs to point to cumin1003

https://gerrit.wikimedia.org/r/1204375

gerritbot added a comment.Nov 13 2025, 8:28 AM

Change #1204797 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove cumin1002 from list of Cumin masters

https://gerrit.wikimedia.org/r/1204797

gerritbot added a comment.Nov 13 2025, 8:40 AM

Change #1204799 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove cumin1002 from mysql root list

https://gerrit.wikimedia.org/r/1204799

gerritbot added a comment.Nov 13 2025, 8:52 AM

Change #1204799 merged by Muehlenhoff:

[operations/puppet@production] Remove cumin1002 from mysql root list

https://gerrit.wikimedia.org/r/1204799

Marostegui closed subtask T409929: Remove cumin1002 grants from production as Resolved.Nov 13 2025, 9:57 AM
gerritbot added a comment.Nov 13 2025, 10:25 AM

Change #1204622 merged by Muehlenhoff:

[operations/puppet@production] Remove cumin1002 as Homer git peer

https://gerrit.wikimedia.org/r/1204622

gerritbot added a comment.Nov 13 2025, 11:07 AM

Change #1204818 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Setup cumin1002 to insetup

https://gerrit.wikimedia.org/r/1204818

gerritbot added a comment.Nov 13 2025, 11:20 AM

Change #1204818 merged by Muehlenhoff:

[operations/puppet@production] Setup cumin1002 to insetup

https://gerrit.wikimedia.org/r/1204818

MoritzMuehlenhoff updated the task description. (Show Details)Nov 13 2025, 11:24 AM
gerritbot added a comment.Nov 13 2025, 11:26 AM

Change #1204628 merged by Marostegui:

[operations/puppet@production] wmf_root_client.pp: Remove cumin1002

https://gerrit.wikimedia.org/r/1204628

gerritbot added a comment.Nov 13 2025, 2:44 PM

Change #1204797 merged by Muehlenhoff:

[operations/puppet@production] Remove cumin1002 from list of Cumin masters

https://gerrit.wikimedia.org/r/1204797

gerritbot added a comment.Nov 13 2025, 2:51 PM

Change #1204368 merged by Muehlenhoff:

[operations/puppet@production] Enable nftables on cluster::management on the role level

https://gerrit.wikimedia.org/r/1204368

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2025, 3:31 PM
bking closed subtask T390860: Elasticsearch dependency upgrade in spicerack as Invalid.Nov 14 2025, 5:59 PM
bking reopened subtask T390860: Elasticsearch dependency upgrade in spicerack as In Progress.Nov 14 2025, 6:12 PM
bking closed subtask T390860: Elasticsearch dependency upgrade in spicerack as Resolved.Nov 17 2025, 11:18 PM
MoritzMuehlenhoff added a comment.Nov 21 2025, 8:16 AM

@Volans made a copy of old the Spicerack/Cumin logs , they are available in /var/log/cumin100[12] on cumin1003 in case anyone needs them.

MoritzMuehlenhoff updated the task description. (Show Details)Nov 21 2025, 8:16 AM
ops-monitoring-bot added a comment.Nov 21 2025, 9:16 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: cumin1002.eqiad.wmnet

  • cumin1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
MoritzMuehlenhoff closed this task as Resolved.Nov 21 2025, 9:18 AM
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff updated the task description. (Show Details)

All done

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits