⚓ T384595 Upgrade Collab hosts to Bookworm

Subject	Repo	Branch	Lines +/-
aptrepo: add gitlab package for bookworm	operations/puppet	production	+11 -0
devtools: update hiera config for new bookworm hosts	operations/puppet	production	+23 -70
gitlab-runner: upgrade default image to bookworm on Trusted Runners	operations/puppet	production	+1 -1
gitlab-runner: upgrade default image to bookworm	operations/puppet	production	+2 -2
ci: switch jenkins deployment method on contint to scap	operations/puppet	production	+10 -9
jenkins: fix puppet error, systemd override requires systemd service	operations/puppet	production	+5 -4
jenkins: ensure systemd service dir exists before override	operations/puppet	production	+9 -4
releases: invert use_scap3_deployment for jenkins	operations/puppet	production	+8 -1
site: revert releases to production role	operations/puppet	production	+1 -5
releases: add force puppet 7 hiera	operations/puppet	production	+1 -0
site: revert releases2003 to insetup	operations/puppet	production	+5 -1
site: add future bookworm doc hosts with insetup role	operations/puppet	production	+4 -0
rt: sunsetting caching	operations/puppet	production	+2 -2
rt: discarding modules about request tracker	operations/puppet	production	+0 -385
httpbb: remove tests for rt.wikimedia.org	operations/puppet	production	+0 -16
rt: removing email configurations	operations/puppet	production	+0 -3
installserver: remove moscovium	operations/puppet	production	+1 -1
requesttracker: remove blackbox monitoring	operations/puppet	production	+0 -10

Title	Reference	Author	Source Branch	Dest Branch
Bump CI image to bookworm (followup)	repos/releng/gitlab-trusted-runner!128	dancy	main-I8148ef92cba017f8657b83dd62042e5ad7c93281	main
Bump CI image to bookworm	repos/releng/gitlab-trusted-runner!127	dancy	main-I921bcef9c2899e171874f63da63909aa84493821	main
Bump CI image to bookworm	repos/releng/gitlab-trusted-runner!126	dancy	main-I921bcef9c2899e171874f63da63909aa84493821	main
Draft: bump CI image to bookworm	repos/releng/gitlab-trusted-runner!125	jelto	ci-bookworm	main

Change #1136765 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: ensure systemd service dir exists before override

https://gerrit.wikimedia.org/r/1136765

Change #1135994 abandoned by Dzahn:

[operations/puppet@production] jenkins: fix puppet error, systemd override requires systemd service

Reason:

continue at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1136765

https://gerrit.wikimedia.org/r/1135994

Change #1136765 merged by Dzahn:

[operations/puppet@production] jenkins: ensure systemd service dir exists before override

https://gerrit.wikimedia.org/r/1136765

Dzahn added a subtask: T391590: PuppetFailure - releases2003.Apr 15 2025, 11:26 PM

Dzahn closed subtask T391590: PuppetFailure - releases2003 as Resolved.

Icinga downtime and Alertmanager silence (ID=fd8d0c80-cd7d-469a-8105-f184bc41f4f2) set by aokoth@cumin1002 for 2 days, 0:00:00 on 1 host(s) and their services with reason: Bookworm Re-image

aphlict2001.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by aokoth@cumin1002 for host aphlict2001.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by aokoth@cumin1002 for host aphlict2001.codfw.wmnet with OS bookworm completed:

aphlict2001 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504161211_aokoth_152128_aphlict2001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Dzahn removed a subtask: T391590: PuppetFailure - releases2003.Apr 16 2025, 5:03 PM

miscweb2003/miscweb1003 can probably be removed from the list since we are not planning to keep running them (soonish).

Arnoldokoth updated the task description. (Show Details)Apr 16 2025, 5:13 PM

LSobanski raised the priority of this task from Low to Medium.Apr 17 2025, 10:47 AM

Dzahn closed subtask T385777: moscovium and RT sunset as Resolved.Apr 18 2025, 9:36 PM

MoritzMuehlenhoff updated the task description. (Show Details)Apr 23 2025, 9:22 AM

MoritzMuehlenhoff added a subtask: T392464: Upgrade Gerrit hosts from Bullseye to Bookworm.

Icinga downtime and Alertmanager silence (ID=89d0151d-5211-42af-a90d-110fc29b52da) set by aokoth@cumin1002 for 2 days, 0:00:00 on 1 host(s) and their services with reason: Bookworm Re-image

aphlict2001.codfw.wmnet

Dzahn added a subtask: T372804: setup gerrit2003 with gerrit service (gerrit on bookworm).Apr 24 2025, 12:32 AM

Icinga downtime and Alertmanager silence (ID=af8c7478-bbf4-41a4-acbf-2af5fb14ee0e) set by aokoth@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Bookworm Reboot

aphlict2001.codfw.wmnet

Arnoldokoth updated the task description. (Show Details)Apr 29 2025, 7:45 PM

Arnoldokoth closed subtask T392128: upgrade aphlict hosts to bookworm as Resolved.May 7 2025, 6:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by aokoth@cumin1002 for host doc1004.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by aokoth@cumin1002 for host doc1004.eqiad.wmnet with OS bookworm completed:

doc1004 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505201905_aokoth_3979640_doc1004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul2002.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul2002.codfw.wmnet with OS bookworm completed:

zuul2002 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505202021_dzahn_3988586_zuul2002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul2003.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul2003.codfw.wmnet with OS bookworm completed:

zuul2003 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505202141_dzahn_4003773_zuul2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

LSobanski updated the task description. (Show Details)Jun 2 2025, 3:19 PM

LSobanski updated the task description. (Show Details)Jun 2 2025, 3:21 PM

LSobanski updated the task description. (Show Details)Jun 2 2025, 3:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host gitlab-runner1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host gitlab-runner1002.eqiad.wmnet with OS bookworm completed:

gitlab-runner1002 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506030925_jelto_3346180_gitlab-runner1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Jelto updated the task description. (Show Details)Jun 3 2025, 9:52 AM

I reimaged gitlab-runner1002 to bookworm. So far the runner looks good and healthy. But I'll monitor the jobs and metrics.

I'll also replace the devtools test-runners with bookworm VMs and one of the VMs in gitlab-runner WMCS project (runner-1021).

hashar reopened subtask T372804: setup gerrit2003 with gerrit service (gerrit on bookworm) as Open.Jun 3 2025, 3:04 PM

LSobanski updated the task description. (Show Details)Jun 3 2025, 3:11 PM

• bd808 updated the task description. (Show Details)Jun 3 2025, 9:00 PM

LSobanski updated the task description. (Show Details)Jun 4 2025, 5:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host gitlab-runner1003.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host gitlab-runner1003.eqiad.wmnet with OS bookworm completed:

gitlab-runner1003 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506160904_jelto_2258786_gitlab-runner1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Jelto updated the task description. (Show Details)Jun 16 2025, 9:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host gitlab-runner1004.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host gitlab-runner1004.eqiad.wmnet with OS bookworm completed:

gitlab-runner1004 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506161157_jelto_2430088_gitlab-runner1004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Jelto updated the task description. (Show Details)Jun 16 2025, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host gitlab-runner2002.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host gitlab-runner2002.codfw.wmnet with OS bookworm completed:

gitlab-runner2002 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506161345_jelto_2542161_gitlab-runner2002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Jelto updated the task description. (Show Details)Jun 16 2025, 2:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host gitlab-runner2003.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host gitlab-runner2003.codfw.wmnet with OS bookworm completed:

gitlab-runner2003 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506170748_jelto_3201448_gitlab-runner2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Jelto updated the task description. (Show Details)Jun 17 2025, 8:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host gitlab-runner2004.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host gitlab-runner2004.codfw.wmnet with OS bookworm completed:

gitlab-runner2004 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202506170833_jelto_3209289_gitlab-runner2004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

All Trusted GitLab runners were upgraded to bookworm.

Change #1160119 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab-runner: upgrade default image to bookworm

https://gerrit.wikimedia.org/r/1160119

Change #1160120 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab-runner: upgrade default image to bookworm on Trusted Runners

https://gerrit.wikimedia.org/r/1160120

doc2002 and doc1003 are already replaced with doc2003 and doc1004 by @Arnoldokoth

So they just need decom at this point.

Change #1160119 merged by Jelto:

[operations/puppet@production] gitlab-runner: upgrade default image to bookworm

https://gerrit.wikimedia.org/r/1160119

In T377889 we now managed to deploy phabricator with scap now..and installed PHP 8.

This let's us confirm if Phabricator works on bookworm and with PHP8.. which should unblock the upgrades of the phab hosts here.

Change #1160120 merged by Jelto:

[operations/puppet@production] gitlab-runner: upgrade default image to bookworm on Trusted Runners

https://gerrit.wikimedia.org/r/1160120

Change #1164222 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] devtools: update hiera config for new bookworm hosts

https://gerrit.wikimedia.org/r/1164222

Change #1164222 merged by Jelto:

[operations/puppet@production] devtools: update hiera config for new bookworm hosts

https://gerrit.wikimedia.org/r/1164222

All devtools runners were upgraded to bookworm. I also updated one of the WMCS shared runners runner-1031.gitlab-runners.eqiad1.wikimedia.cloud to bookwom. I'll covert the others next week. Meanwhile I also updated the documentation for adding new WMCS runners.

The new WMCS runners also have a new flavor with a second 90GB disk, which should give /var/lib/docker a bit more space.

Scott_French mentioned this in T378128: Upgrade httpd images to bullseye or bookworm.Jun 30 2025, 2:22 PM

Arnoldokoth closed subtask T392130: upgrade doc hosts to bookworm as Resolved.Jul 1 2025, 8:34 AM

Jelto mentioned this in T398628: Upgrade WMCS Shared GitLab Runners to bookworm.Jul 3 2025, 1:28 PM

Jelto closed subtask T398628: Upgrade WMCS Shared GitLab Runners to bookworm as Resolved.Jul 7 2025, 1:49 PM

doc hosts checked off the list, linked to resolved ticket for them

Change #1167871 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] aptrepo: add gitlab package for bookworm

https://gerrit.wikimedia.org/r/1167871

Change #1167871 merged by Dzahn:

[operations/puppet@production] aptrepo: add gitlab package for bookworm

https://gerrit.wikimedia.org/r/1167871

gitlab-ce is available for bookworm now:

jelto@apt1002:~$ sudo -i reprepro lsbycomponent gitlab-ce
gitlab-ce | 18.0.4-ce.0 | bullseye-wikimedia | thirdparty/gitlab-bullseye | amd64
gitlab-ce | 18.0.4-ce.0 | bookworm-wikimedia | thirdparty/gitlab-bookworm | amd64

For the next steps of the upgrade I'll add a subtask (the upgrade also involves a switchover between the GitLab hosts).

Jelto updated the task description. (Show Details)Jul 14 2025, 9:47 AM

Jelto updated the task description. (Show Details)Jul 15 2025, 11:47 AM

Jelto updated the task description. (Show Details)Jul 18 2025, 7:31 AM

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/125

bump CI image to bookworm

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/126

Bump CI image to bookworm

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/126

Bump CI image to bookworm

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/127

Bump CI image to bookworm

dancy closed https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/127

Bump CI image to bookworm

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/128

Bump CI image to bookworm (followup)

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/128

Bump CI image to bookworm (followup)

dancy closed https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/125

Draft: bump CI image to bookworm

Jelto updated the task description. (Show Details)Aug 5 2025, 2:03 PM

Jelto closed subtask T399306: upgrade gitlab hosts to bookworm as Resolved.Aug 6 2025, 7:19 AM

Dzahn updated the task description. (Show Details)Aug 12 2025, 10:33 PM

Dzahn updated the task description. (Show Details)

Dzahn changed the status of subtask T372804: setup gerrit2003 with gerrit service (gerrit on bookworm) from Open to In Progress.Sep 26 2025, 4:00 PM

Dzahn changed the status of subtask T392127: upgrade releases hosts to bookworm from Open to In Progress.Nov 13 2025, 9:07 PM

Dzahn closed subtask T392127: upgrade releases hosts to bookworm as Resolved.Wed, Nov 19, 8:06 PM

Dzahn updated the task description. (Show Details)

Dzahn closed subtask T372804: setup gerrit2003 with gerrit service (gerrit on bookworm) as Resolved.Thu, Dec 4, 11:23 PM

Status	Subtype	Assigned	Task
Open		None	T407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
Open		None	T407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
Open		None	T387831 Standardize failover procedures for Collab services
Resolved		None	T393239 ProbeDown
Open		None	T392448 Upgrade to Gerrit 3.12
Open		None	T379714 Upgrade to Gerrit 3.11
Open		None	T392465 Switch Gerrit from Java 17 to Java 21
Open		None	T384595 Upgrade Collab hosts to Bookworm
Resolved		Dzahn	T385777 moscovium and RT sunset
Resolved		jcrespo	T385901 mysqldump of RT database
Duplicate		Marostegui	T388432 Drop RT database from m1
Resolved		Dzahn	T336320 scrape RT ticket HTML files
Resolved		Marostegui	T388437 drop RT database and mysql grants
Resolved		Dzahn	T392127 upgrade releases hosts to bookworm
Resolved		Dzahn	T391590 PuppetFailure - releases2003
Resolved		Dzahn	T410422 SystemdUnitFailed - rsync-srv-org-wikimedia-releases
Resolved		Arnoldokoth	T392128 upgrade aphlict hosts to bookworm
Resolved		Dzahn	T392480 PuppetDisabled (aphlict2001)
Resolved		Arnoldokoth	T392130 upgrade doc hosts to bookworm
Resolved		Arnoldokoth	T395275 SystemdUnitFailed - doc1003 - rsync-doc-host-data-sync
Open		None	T392464 Upgrade Gerrit hosts from Bullseye to Bookworm
In Progress		ABran-WMF	T387833 Gerrit switchover process
Open		None	T388507 Investigate missing commit from old changes
Resolved		ABran-WMF	T260666 Create a cookbook to automate gerrit's switchover
Resolved		ABran-WMF	T393050 ProbeDown
Resolved		MatthewVernon	T392186 Grant Gerrit admin to arnaudb
Resolved		LSobanski	T393034 Investigate out of date refs following gerrit switchover
Resolved		ABran-WMF	T395440 Gerrit read-only plugin test
Open		ABran-WMF	T406334 Gerrit switchover between secondary instances
Resolved		LSobanski	T406482 SystemdUnitFailed (jenkins.service on contint1002)
Resolved		hashar	T406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
Resolved		ABran-WMF	T411583 Gerrit backups are growing
Resolved		hashar	T406856 Reduce size of analytics/superset/deploy.git Gerrit repo
			Restricted Task
Open		ABran-WMF	T412779 Gerrit failover process
Resolved		Dzahn	T372804 setup gerrit2003 with gerrit service (gerrit on bookworm)
In Progress		ABran-WMF	T338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
Resolved		Dzahn	T379213 PuppetFailure - gerrit2003
Open		None	T333143 Move Gerrit data out of root partition
Resolved		Marostegui	T388785 SSH on gerrit2003 is CRITICAL: CRITICAL
Resolved		Dzahn	T374899 PuppetFailure - gerrit2003
Open		None	T257317 scap deploy --init on deployment server fails on first puppet run
Resolved		Jelto	T398628 Upgrade WMCS Shared GitLab Runners to bookworm
Resolved		Jelto	T399306 upgrade gitlab hosts to bookworm
Open		None	T399714 GitLab partman config is unreliable
Resolved		ABran-WMF	T400121 Gitlab-replica switchover (gitlab1004 → gitlab1003)
Resolved		Jelto	T400252 Gitlab switchover (gitlab2002 → gitlab1004)
Resolved		None	T400663 SystemdUnitFailed (config-backup.service on gitlab2002)
Resolved		Jelto	T400695 Remove home_page_url from gitlab-settings
Resolved		Jelto	T400789 PuppetFailure (releases hosts)
Resolved		Jelto	T400816 SystemdUnitFailed (prometheus-nft-throttling-denylist)
Resolved		ABran-WMF	T400971 Troubleshoot GitLab nftables throttling after switchover
Resolved	BUG REPORT	Jelto	T401535 Uploading artifacts does not work anymore
			Restricted Task
Resolved		Dzahn	T402847 ProbeDown - gerrit1003
Resolved		ABran-WMF	T403347 ProbeDown
Resolved		ABran-WMF	T403963 ProbeDown - http_gerrit_tls_ip4 gerrit2003
Resolved		ABran-WMF	T404070 ProbeDown - http_gerrit_tls_ip4 gerrit2003.
Resolved		Dzahn	T401191 SystemdUnitFailed - wmf_auto_restart_ssh-gitlab.service on gitlab2002
Resolved		Jelto	T401261 SystemdUnitFailed (rsync-data-backup-gitlab2002)

Upgrade Collab hosts to Bookworm
Open, MediumPublic
Actions

Description

Details

Related Objects
Search...

Event Timeline

	LSobanski
	Jan 23 2025, 12:55 PM

Upgrade Collab hosts to BookwormOpen, MediumPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Upgrade Collab hosts to Bookworm
Open, MediumPublic
Actions

Related Objects
Search...