Page MenuHomePhabricator
Create Task
Maniphest T417246

Reimage gerrit1003
Closed, ResolvedPublic
Actions

Description

Following up on T387833: Gerrit switchover process we'll need to reimage gerrit1003 on Debian 12.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gerrit: fix gerrit1003 ssh fingerprintoperations/puppetproduction+3 -1
gerrit: gerrit-spare lfs-sync enableoperations/puppetproduction+1 -1
gerrit: resume replication on gerrit-spareoperations/puppetproduction+3 -3
java: update puppet certificateoperations/puppetproduction+1 -1
backup: Temporarily ignore backup job failures from gerrit1003operations/puppetproduction+3 -0
gerrit: change gerrit1003 roleoperations/puppetproduction+1 -5
gerrit: move gerrit1003 to insetupoperations/puppetproduction+5 -1
gerrit: move gerrit1003 to insetupoperations/puppetproduction+5 -1
gerrit: change system user for gerrit1003operations/puppetproduction+1 -1
gerrit: change daemon user for gerrit1003operations/puppetproduction+9 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

ABran-WMF created this task.Feb 12 2026, 9:11 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 12 2026, 9:11 AM
ABran-WMF triaged this task as High priority.Feb 12 2026, 9:13 AM
ABran-WMF moved this task from Incoming to Backlog on the collaboration-services board.
ABran-WMF moved this task from Backlog to Work in Progress on the collaboration-services board.Feb 18 2026, 9:18 AM
ops-monitoring-bot added a comment.Feb 18 2026, 9:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1003 for host gerrit1003.wikimedia.org with OS bookworm

gerritbot added a comment.Feb 18 2026, 9:30 AM

Change #1240219 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: change daemon user for gerrit1003

https://gerrit.wikimedia.org/r/1240219

gerritbot added a project: Patch-For-Review.Feb 18 2026, 9:30 AM

Change #1240219 merged by Arnaudb:

[operations/puppet@production] gerrit: change daemon user for gerrit1003

https://gerrit.wikimedia.org/r/1240219

ABran-WMF added a comment.Feb 18 2026, 10:32 AM

I missed a spot in puppet:

$ grep gerrit /etc/passwd
27:gerrit2:x:925:925::/var/lib/gerrit2:/bin/bash
gerritbot added a comment.Feb 18 2026, 10:35 AM

Change #1240230 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: change system user for gerrit1003

https://gerrit.wikimedia.org/r/1240230

ops-monitoring-bot added a comment.Feb 18 2026, 10:41 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1003 for host gerrit1003.wikimedia.org with OS bookworm executed with errors:

  • gerrit1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602180953_arnaudb_2362553_gerrit1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console gerrit1003.wikimedia.org" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Feb 18 2026, 10:52 AM

Change #1240230 merged by Arnaudb:

[operations/puppet@production] gerrit: change system user for gerrit1003

https://gerrit.wikimedia.org/r/1240230

ops-monitoring-bot added a comment.Feb 18 2026, 10:56 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1003 for host gerrit1003.wikimedia.org with OS bookworm

ABran-WMF added a comment.Feb 18 2026, 11:39 AM

missed spot cleaned up:

arnaudb@gerrit1003:~ $ grep gerrit /etc/passwd
27:gerrit:x:925:925::/var/lib/gerrit:/bin/bash
ABran-WMF created subtask T417767: scap error on Gerrit first setup.Feb 18 2026, 11:58 AM
ABran-WMF mentioned this in T417767: scap error on Gerrit first setup.
Stashbot added a comment.Feb 18 2026, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-18T13:24:56Z] <arnaudb@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on gerrit1003.wikimedia.org with reason: T417246

Dzahn added a comment.Feb 18 2026, 3:13 PM

I would not expect a reimage to work while the puppet role is applied. Recommend moving to "insetup" role, reimage and then apply gerrit prod role again.

Errors like the scap error expected and making it work is one thing but trying to "make it work on 1st puppet run" is another level. That would take a lot and end of the day it only matters once every few years.

gerritbot added a comment.Feb 18 2026, 3:16 PM

Change #1240306 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: move gerrit1003 to insetup

https://gerrit.wikimedia.org/r/1240306

gerritbot added a comment.Feb 18 2026, 3:19 PM

Change #1240306 merged by Arnaudb:

[operations/puppet@production] gerrit: move gerrit1003 to insetup

https://gerrit.wikimedia.org/r/1240306

gerritbot added a comment.Feb 18 2026, 3:26 PM

Change #1240314 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: move gerrit1003 to insetup

https://gerrit.wikimedia.org/r/1240314

gerritbot added a comment.Feb 18 2026, 4:11 PM

Change #1240314 merged by Arnaudb:

[operations/puppet@production] gerrit: move gerrit1003 to insetup

https://gerrit.wikimedia.org/r/1240314

ops-monitoring-bot added a comment.Feb 18 2026, 4:12 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1003 for host gerrit1003.wikimedia.org with OS bookworm executed with errors:

  • gerrit1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602181119_arnaudb_2374336_gerrit1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console gerrit1003.wikimedia.org" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 18 2026, 4:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1003 for host gerrit1003.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Feb 18 2026, 4:53 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1003 for host gerrit1003.wikimedia.org with OS bookworm completed:

  • gerrit1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602181634_arnaudb_2665920_gerrit1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Feb 19 2026, 6:26 AM

Change #1240471 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: change gerrit1003 role

https://gerrit.wikimedia.org/r/1240471

gerritbot added a comment.Feb 19 2026, 6:31 AM

Change #1240471 merged by Arnaudb:

[operations/puppet@production] gerrit: change gerrit1003 role

https://gerrit.wikimedia.org/r/1240471

gerritbot added a comment.Feb 19 2026, 7:08 AM

Change #1240477 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] java: update puppet certificate

https://gerrit.wikimedia.org/r/1240477

ABran-WMF closed subtask T417767: scap error on Gerrit first setup as Resolved.Feb 19 2026, 9:28 AM
gerritbot added a comment.Feb 19 2026, 11:57 AM

Change #1240648 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] backup: Temporarilly ignore backup job failures from gerrit1003

https://gerrit.wikimedia.org/r/1240648

gerritbot added a comment.Feb 19 2026, 12:00 PM

Change #1240648 merged by Jcrespo:

[operations/puppet@production] backup: Temporarily ignore backup job failures from gerrit1003

https://gerrit.wikimedia.org/r/1240648

gerritbot added a comment.Feb 19 2026, 1:01 PM

Change #1240477 merged by Arnaudb:

[operations/puppet@production] java: update puppet certificate

https://gerrit.wikimedia.org/r/1240477

gerritbot added a comment.Feb 19 2026, 1:22 PM

Change #1240689 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: resume replication on gerrit-spare

https://gerrit.wikimedia.org/r/1240689

ABran-WMF changed the task status from Open to Stalled.Feb 19 2026, 1:57 PM
ABran-WMF moved this task from Work in Progress to Awaiting Input on the collaboration-services board.

moving to stalled until https://gerrit.wikimedia.org/r/c/1240689 is merged

gerritbot added a comment.Feb 20 2026, 7:29 AM

Change #1240689 merged by Arnaudb:

[operations/puppet@production] gerrit: resume replication on gerrit-spare

https://gerrit.wikimedia.org/r/1240689

ABran-WMF changed the task status from Stalled to In Progress.Feb 20 2026, 7:41 AM

Once gerrit1003 has been repooled, replication_log raised the following issue:

[2026-02-20 07:38:34,958] Cannot replicate to gerrit@gerrit1003.wikimedia.org:/srv/gerrit/git/design/codex-php.git [CONTEXT pushOneId="b439c1e0" ]
org.eclipse.jgit.errors.TransportException: gerrit@gerrit1003.wikimedia.org:/srv/gerrit/git/design/codex-php.git: [ssh-connection]: Failed (UnsupportedCredentialItem) to execute: ssh://gerrit@gerrit1003.wikimedia.org:22: org.eclipse.jgit.transport.CredentialItem$YesNoType:Accept this key and continue connecting all the same?
ABran-WMF added a subtask: T417968: ProbeDown.Feb 20 2026, 7:44 AM
ABran-WMF added a subtask: T417966: GerritHAProxyBackendUnavailable.
ABran-WMF closed subtask T417966: GerritHAProxyBackendUnavailable as Resolved.
ABran-WMF moved this task from Awaiting Input to Work in Progress on the collaboration-services board.Feb 20 2026, 7:46 AM
gerritbot added a comment.Feb 20 2026, 7:49 AM

Change #1240846 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: gerrit-spare lfs-sync enable

https://gerrit.wikimedia.org/r/1240846

ABran-WMF added a comment.Feb 20 2026, 7:56 AM

after a fix ad-hoc:

gerrit@gerrit2003:~$ diff ~/.ssh/known_hosts ~/.ssh/known_hosts.bak
3a4
> gerrit1003.wikimedia.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbcAskAyVZGB+o358NsGwsiiSc/Dd2nZm1pPNAXOFpXkiOSQ7K6eRzThJRV82VUE/ypNXAGhgwIpW3HOxpgb03FUTetZHKseA2Q3oYJ1MzVLCj+C9QUWPRA5FsT3R8f/1+fDNL00v+X3unFmQ2huxSdeL4jKHpO9GxMSF50cx6qseJGmdv+Ry75vbi+CNte9kpsUbllknBGNH4/yhxXjIypPiZQWPT1tq/YZsCtvO4zn+4NXk7016jeZTEn/SLXiqeNDiTD01HNkb8S+8EZym/hD0tUTjW8i7xYcqPBIKEekGJvdbi8nFl41Vy/jpXZkjgPUL1y9124fGSS8jyMhC/
5,7d5
< gerrit1003.wikimedia.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCKt0oPPdZcPbx/aTRCBbBEtSUuXXU1Vgbq7tOcZlSWhcqEJ7SDrXl8lduYh+7ivcQdwYo/NszXv0jyO6n+mObd6Wu3/Pzkw5/aB9xnWmmNut+xDn21MqyaGuJ6kJjJsrvm4VG+2GJ5DiTigwSDMLwRoKUNCPAV+RITM7tCIAb/7Kf1PkCPQDAP+Zhvbe0ISEojJ4/Sw/aMDCyeP42R+EIknaFg610DP0sN/Imy2LmMJf+o8xzWg48NWnWjOMRE9q8ktxiawXNksZaeDCMJdnrHNKLWl3vgSqVrIqspIP45OuVuN/3NSrRZ4KbR3ctB87pjU/64CySXOAFbG7c8uHmL
< gerrit1003.wikimedia.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFIw3OVtnjX1xBTN7JWXxbr79TJuopjxQ1vtca1uqTP5Qjjphr5SHxk70OUVfmnEWj+cfvLp/e82XUj/2vsUv9c=
< gerrit1003.wikimedia.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFHUgsuuPPnGS/9zs9UzLxput0iASwIt8/x9TZDj/4Zf
gerritbot added a comment.Feb 20 2026, 7:58 AM

Change #1240848 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: fix gerrit1003 ssh fingerprint

https://gerrit.wikimedia.org/r/1240848

gerritbot added a comment.Feb 20 2026, 8:00 AM

Change #1240846 merged by Arnaudb:

[operations/puppet@production] gerrit: gerrit-spare lfs-sync enable

https://gerrit.wikimedia.org/r/1240846

gerritbot added a comment.Feb 20 2026, 8:01 AM

Change #1240848 merged by Arnaudb:

[operations/puppet@production] gerrit: fix gerrit1003 ssh fingerprint

https://gerrit.wikimedia.org/r/1240848

ABran-WMF closed this task as Resolved.Feb 20 2026, 8:11 AM

This can be considered resolved, replication is running

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits