Page MenuHomePhabricator
Create Task
Maniphest T392464

Upgrade Gerrit hosts from Bullseye to Bookworm
Closed, ResolvedPublic
Actions

Description

Bookworm → Java 21 → Gerrit 3.11 → Gerrit 3.12

The Gerrit hosts are currently running Debian Bullseye and the application runs under Java 17. Gerrit 3.12 will drop support for Java 17 and require Java 21.

Both Debian Bullseye and Bookworm come with Java 17.
For Debian Bookworm we have component/jdk21 which let us upgrade to Java 21 (T392448#10759611).

Therefore we should reimage the Gerrit hosts from Bullseye to Bookworm, keeping java 17.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
 OpenNoneT407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
 ResolvedNoneT393239 ProbeDown
Restricted Task
 OpenNoneT392448 Upgrade to Gerrit 3.12
 OpenNoneT379714 Upgrade to Gerrit 3.11
 OpenNoneT392465 Switch Gerrit from Java 17 to Java 21
 OpenNoneT384595 Upgrade Collab hosts to Bookworm
 ResolvedABran-WMFT392464 Upgrade Gerrit hosts from Bullseye to Bookworm
 ResolvedABran-WMFT387833 Gerrit switchover process
 OpenNoneT388507 Investigate missing commit from old changes
 ResolvedABran-WMFT260666 Create a cookbook to automate gerrit's switchover
 ResolvedABran-WMFT393050 ProbeDown
 ResolvedMatthewVernonT392186 Grant Gerrit admin to arnaudb
 ResolvedLSobanskiT393034 Investigate out of date refs following gerrit switchover
 ResolvedABran-WMFT395440 Gerrit read-only plugin test
 ResolvedABran-WMFT406334 Gerrit switchover between secondary instances
 ResolvedLSobanskiT406482 SystemdUnitFailed (jenkins.service on contint1002)
 ResolvedhasharT406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
 ResolvedABran-WMFT411583 Gerrit backups are growing
 ResolvedhasharT406856 Reduce size of analytics/superset/deploy.git Gerrit repo
 ResolvedABran-WMFT407200 Gerrit is down
 ResolvedABran-WMFT412779 Gerrit failover process
 ResolvedABran-WMFT417246 Reimage gerrit1003
 ResolveddancyT417767 scap error on Gerrit first setup
 ResolvedABran-WMFT417777 SystemdUnitFailed - apache2.service on gerrit1003:9100
 DuplicateNoneT417968 ProbeDown
 DuplicateABran-WMFT417966 GerritHAProxyBackendUnavailable
 ResolvedABran-WMFT417247 Reimage gerrit2002
 ResolvedABran-WMFT418242 SystemdUnitFailed - rsync-lfs_replica_sync.service on gerrit2002:9100
 ResolvedABran-WMFT418264 Fix gerrit-restart cookbook
 ResolvedABran-WMFT418266 GerritHAProxyServiceUnavailable
 ResolvedABran-WMFT418240 GerritReplicationUnavailable
 ResolvedABran-WMFT417279 Harmonize DNS on all gerrit instances
 ResolvedABran-WMFT417565 GerritHAProxyBackendUnavailable
 ResolvedABran-WMFT417566 GerritHAProxyServiceUnavailable
 ResolvedDzahnT372804 setup gerrit2003 with gerrit service (gerrit on bookworm)
 ResolvedABran-WMFT338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
 ResolvedDzahnT379213 PuppetFailure - gerrit2003
 OpenNoneT417642 Rename gerrit2 Gerrit user to gerrit in the administrator group
 ResolvedJeltoT333143 Move Gerrit data out of root partition
 OpenNoneT424172 gerrit: document replication re-runs to Github
 ResolvedJeltoT424924 GerritHAProxyBackendUnavailable
 ResolvedMarosteguiT388785 SSH on gerrit2003 is CRITICAL: CRITICAL
 ResolvedDzahnT374899 PuppetFailure - gerrit2003
 OpenNoneT257317 scap deploy --init on deployment server fails on first puppet run
Restricted Task
 ResolvedDzahnT402847 ProbeDown - gerrit1003
 ResolvedABran-WMFT403347 ProbeDown
 ResolvedABran-WMFT403963 ProbeDown - http_gerrit_tls_ip4 gerrit2003
 ResolvedABran-WMFT404070 ProbeDown - http_gerrit_tls_ip4 gerrit2003.

Event Timeline

hashar created this task.Apr 23 2025, 9:11 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 23 2025, 9:11 AM
hashar mentioned this in T392448: Upgrade to Gerrit 3.12.Apr 23 2025, 9:14 AM
hashar mentioned this in T392465: Switch Gerrit from Java 17 to Java 21.Apr 23 2025, 9:18 AM
MoritzMuehlenhoff mentioned this in T384595: Upgrade Collab hosts to Bookworm.Apr 23 2025, 9:21 AM
MoritzMuehlenhoff added a parent task: T384595: Upgrade Collab hosts to Bookworm.
MoritzMuehlenhoff subscribed.
hashar added a parent task: T392465: Switch Gerrit from Java 17 to Java 21.Apr 23 2025, 9:27 AM
Dzahn subscribed.Apr 24 2025, 12:30 AM

Yea.. this is what gerrit2003 was made for (T372804) ...

Dzahn added a comment.Apr 24 2025, 12:42 AM

That's also where T338470 is fixed already.

We have had that for months yet there didn't seem to be interest in testing and switching to it.

But it should happen as part of T387833. Then once 2003 is production we can reimage the other 2 older hosts.

hashar renamed this task from Upgrade Gerrit hosts to Bookworm to Upgrade Gerrit hosts from Bullseye to Bookworm.Apr 28 2025, 12:54 PM
Jelto mentioned this in T400971: Troubleshoot GitLab nftables throttling after switchover.Aug 4 2025, 1:10 PM
hashar added a subtask: T387833: Gerrit switchover process.Aug 10 2025, 12:30 PM
hashar updated the task description. (Show Details)Aug 11 2025, 7:12 AM
Dzahn added a subtask: T372804: setup gerrit2003 with gerrit service (gerrit on bookworm).Aug 12 2025, 10:30 PM

This is mostly a duplicate of T372804 where we already did the things needed to have a gerrit on bookworm.

The difference being just that we started gerrit2003 only and this should cover all 3 hosts.

hashar added a project: collaboration-services.Sep 17 2025, 8:29 PM

Paladox has let me know Gerrit maintainers have announced support for Gerrit 3.10 will stop on November 10th.
https://gerrit-review.googlesource.com/c/homepage/+/510701

We thus need to upgrade the machines to Bookworm to let us switch to Java 21 and then upgrade to 3.11.

I kind of lost track of where we were regarding upgrading the hosts and I realized this task was not even tagged with collaboration-services.

Dzahn added a comment.Sep 17 2025, 9:31 PM

That's because the actual ticket is T372804 which was created 10 months earlier. Most of this was already done back in September 2024 and has been waiting ever since to be tested. It has been part of the whole failover vs standby discussion.

ABran-WMF subscribed.Sep 18 2025, 1:35 PM
hashar added a subtask: Restricted Task.Sep 18 2025, 2:13 PM
hashar added a comment.Sep 18 2025, 2:15 PM

Per @ABran-WMF:

The upgrade to Bookworm upgrade was blocked by the need to fix throttling to deter scrapers, it should resume after https://gerrit.wikimedia.org/r/c/operations/puppet/+/1189386 is merged.

That is for {T402611}

Dzahn changed the status of subtask T372804: setup gerrit2003 with gerrit service (gerrit on bookworm) from Open to In Progress.Sep 26 2025, 4:00 PM
Dzahn moved this task from Incoming to Work in Progress (Tracking tasks) on the collaboration-services board.Sep 26 2025, 7:21 PM
Dzahn closed subtask T372804: setup gerrit2003 with gerrit service (gerrit on bookworm) as Resolved.Dec 4 2025, 11:23 PM
ABran-WMF closed subtask T387833: Gerrit switchover process as Resolved.Feb 16 2026, 2:57 PM
Dzahn added a comment.Feb 18 2026, 7:04 PM

we should reimage the Gerrit hosts from Bullseye to Bookworm, keeping java 17.

Arnaud reimaged gerrit1003 to bookworm. Java is not installed yet but the JRE will remain 17.

openjdk-17-jre - OpenJDK Java runtime, using Hotspot JIT
Paladox subscribed.Feb 21 2026, 4:54 PM
Dzahn added a comment.Mar 5 2026, 3:18 PM

All gerrit hosts are on bookworm. Thanks to @ABran-WMF .

I think this can be closed as resolved.

ABran-WMF closed this task as Resolved.Mar 5 2026, 3:21 PM
ABran-WMF claimed this task.

great, thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits