Page MenuHomePhabricator
Create Task
Maniphest T379213

PuppetFailure - gerrit2003
Closed, ResolvedPublic
Actions

Description

Common information

  • alertname: PuppetFailure
  • cluster: misc
  • instance: gerrit2003:9100
  • job: node
  • prometheus: ops
  • severity: critical
  • site: codfw
  • source: prometheus
  • team: collaboration-services

Firing alerts

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
 OpenNoneT407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
Restricted Task
 OpenNoneT392448 Upgrade to Gerrit 3.12
 OpenNoneT379714 Upgrade to Gerrit 3.11
 OpenNoneT392465 Switch Gerrit from Java 17 to Java 21
 OpenNoneT387831 Standardize failover procedures for Collab services
 ResolvedNoneT393239 ProbeDown
Unknown Object (Task)
 ResolvedJhancock.wmT369670 Q1:rack/setup/install gerrit2003
 OpenNoneT384595 Upgrade Collab hosts to Bookworm
 ResolvedABran-WMFT392464 Upgrade Gerrit hosts from Bullseye to Bookworm
 ResolvedABran-WMFT387833 Gerrit switchover process
 OpenNoneT388507 Investigate missing commit from old changes
 ResolvedABran-WMFT260666 Create a cookbook to automate gerrit's switchover
 ResolvedABran-WMFT393050 ProbeDown
 ResolvedMatthewVernonT392186 Grant Gerrit admin to arnaudb
 ResolvedLSobanskiT393034 Investigate out of date refs following gerrit switchover
 ResolvedABran-WMFT395440 Gerrit read-only plugin test
 ResolvedABran-WMFT406334 Gerrit switchover between secondary instances
 ResolvedLSobanskiT406482 SystemdUnitFailed (jenkins.service on contint1002)
 ResolvedhasharT406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
 ResolvedABran-WMFT411583 Gerrit backups are growing
 ResolvedhasharT406856 Reduce size of analytics/superset/deploy.git Gerrit repo
 ResolvedABran-WMFT407200 Gerrit is down
 ResolvedABran-WMFT412779 Gerrit failover process
 ResolvedABran-WMFT417246 Reimage gerrit1003
 ResolveddancyT417767 scap error on Gerrit first setup
 ResolvedABran-WMFT417777 SystemdUnitFailed - apache2.service on gerrit1003:9100
 DuplicateNoneT417968 ProbeDown
 DuplicateABran-WMFT417966 GerritHAProxyBackendUnavailable
 ResolvedABran-WMFT417247 Reimage gerrit2002
 ResolvedABran-WMFT418242 SystemdUnitFailed - rsync-lfs_replica_sync.service on gerrit2002:9100
 ResolvedABran-WMFT418264 Fix gerrit-restart cookbook
 ResolvedABran-WMFT418266 GerritHAProxyServiceUnavailable
 ResolvedABran-WMFT418240 GerritReplicationUnavailable
 ResolvedABran-WMFT417279 Harmonize DNS on all gerrit instances
 ResolvedABran-WMFT417565 GerritHAProxyBackendUnavailable
 ResolvedABran-WMFT417566 GerritHAProxyServiceUnavailable
 ResolvedDzahnT372804 setup gerrit2003 with gerrit service (gerrit on bookworm)
 ResolvedABran-WMFT338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
 ResolvedDzahnT379213 PuppetFailure - gerrit2003
 OpenNoneT417642 Rename gerrit2 Gerrit user to gerrit in the administrator group

Event Timeline

phaultfinder created this task.Nov 7 2024, 12:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 7 2024, 12:44 AM
Dzahn renamed this task from PuppetFailure to PuppetFailure - gerrit2003.EditedNov 7 2024, 3:41 AM
Dzahn claimed this task.
Dzahn subscribed.

I am currently working on setting this host up.

Will close out tomorrow.

LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.Nov 7 2024, 9:31 AM
LSobanski merged a task: T379219: PuppetFailure.
Dzahn merged a task: T379274: PuppetFailure - gerrit2003.Nov 7 2024, 4:28 PM
Dzahn added a comment.Nov 7 2024, 6:01 PM

The reason here was that I deleted /var/lib/gerrit2 (since the new directory is /var/lib/gerrit).

But there are still some dependencies on that directory in puppet and that causes a bunch of dependency issues further down.

Dzahn closed this task as Resolved.Nov 7 2024, 6:03 PM

For now I just did an mkdir /var/lib/gerrit2 to stop the puppet error noise. But the real fix is of course to not depend on that directory anymore on the new host.

That will be part of T338470

Dzahn added a parent task: T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid.Nov 7 2024, 6:04 PM
Dzahn added a parent task: T372804: setup gerrit2003 with gerrit service (gerrit on bookworm).Apr 24 2025, 12:36 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits