PuppetFailure - gerrit2003
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Nov 7 2024, 12:44 AM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on gerrit2003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on gerrit2003:9100

alertname: PuppetFailure
cluster: misc
instance: gerrit2003:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on gerrit2003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on gerrit2003:9100
alertname: PuppetFailure
cluster: misc
instance: gerrit2003:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

Related Objects
Search...

Status	Assigned	Task
Open	None	T407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
Open	None	T407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
Open	None	T392448 Upgrade to Gerrit 3.12
Open	None	T379714 Upgrade to Gerrit 3.11
Open	None	T392465 Switch Gerrit from Java 17 to Java 21
Open	None	T387831 Standardize failover procedures for Collab services
Resolved	None	T393239 ProbeDown
		Unknown Object (Task)
Resolved	Jhancock.wm	T369670 Q1:rack/setup/install gerrit2003
Open	None	T384595 Upgrade Collab hosts to Bookworm
Open	None	T392464 Upgrade Gerrit hosts from Bullseye to Bookworm
In Progress	ABran-WMF	T387833 Gerrit failover process
Open	None	T388507 Investigate missing commit from old changes
Resolved	ABran-WMF	T260666 Create a cookbook to automate gerrit's switchover
Resolved	ABran-WMF	T393050 ProbeDown
Resolved	MatthewVernon	T392186 Grant Gerrit admin to arnaudb
Resolved	LSobanski	T393034 Investigate out of date refs following gerrit switchover
Resolved	ABran-WMF	T395440 Gerrit read-only plugin test
Open	ABran-WMF	T406334 Gerrit switchover between secondary instances
Resolved	LSobanski	T406482 SystemdUnitFailed (jenkins.service on contint1002)
Resolved	hashar	T406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
Resolved	hashar	T406856 Reduce size of analytics/superset/deploy.git Gerrit repo
		Restricted Task
In Progress	Dzahn	T372804 setup gerrit2003 with gerrit service (gerrit on bookworm)
In Progress	ABran-WMF	T338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
Resolved	Dzahn	T379213 PuppetFailure - gerrit2003

Event Timeline

phaultfinder created this task.Nov 7 2024, 12:44 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 7 2024, 12:44 AM

I am currently working on setting this host up.

Will close out tomorrow.

LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.Nov 7 2024, 9:31 AM

LSobanski merged a task: T379219: PuppetFailure.

Dzahn merged a task: T379274: PuppetFailure - gerrit2003.Nov 7 2024, 4:28 PM

The reason here was that I deleted /var/lib/gerrit2 (since the new directory is /var/lib/gerrit).

But there are still some dependencies on that directory in puppet and that causes a bunch of dependency issues further down.

For now I just did an mkdir /var/lib/gerrit2 to stop the puppet error noise. But the real fix is of course to not depend on that directory anymore on the new host.

That will be part of T338470

Dzahn added a parent task: T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid.Nov 7 2024, 6:04 PM

Dzahn added a parent task: T372804: setup gerrit2003 with gerrit service (gerrit on bookworm).Apr 24 2025, 12:36 AM

PuppetFailure - gerrit2003Closed, ResolvedPublicActions