Maniphest T374899

PuppetFailure - gerrit2003
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Sep 17 2024, 2:24 AM

Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on gerrit2003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on gerrit2003:9100

alertname: PuppetFailure
cluster: misc
instance: gerrit2003:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/yOxVDGvWk/puppet
description: Puppet failure on gerrit2003:9100
runbook: https://puppetboard.wikimedia.org/nodes?status=failed
summary: Puppet has failed on gerrit2003:9100
alertname: PuppetFailure
cluster: misc
instance: gerrit2003:9100
job: node
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
		Open		None	T407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
					Restricted Task
		Open		None	T392448 Upgrade to Gerrit 3.12
		Open		None	T379714 Upgrade to Gerrit 3.11
		Open		None	T392465 Switch Gerrit from Java 17 to Java 21
					Unknown Object (Task)
		Resolved		Jhancock.wm	T369670 Q1:rack/setup/install gerrit2003
		Open		None	T384595 Upgrade Collab hosts to Bookworm
		Resolved		ABran-WMF	T392464 Upgrade Gerrit hosts from Bullseye to Bookworm
		Resolved		Dzahn	T372804 setup gerrit2003 with gerrit service (gerrit on bookworm)
		Resolved		Dzahn	T374899 PuppetFailure - gerrit2003

Event Timeline

phaultfinder created this task.Sep 17 2024, 2:24 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2024, 2:24 AM

Caused by testing of the gerrit role on bookwork. Host not in production yet.

This should be fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1073308

Waiting for that to be merged.

Unclear how the issue was fixed on current prod host. Probably with a manual command copying the image around.

Dzahn triaged this task as Low priority.Sep 17 2024, 2:19 PM

Dzahn mentioned this in T374966: PuppetFailure - gerrit2003.Sep 17 2024, 5:57 PM

Dzahn merged a task: T374966: PuppetFailure - gerrit2003.

Dzahn moved this task from Incoming to Work in Progress (Tracking tasks) on the collaboration-services board.Sep 17 2024, 8:34 PM

Dzahn moved this task from Work in Progress (Tracking tasks) to Work in Progress on the collaboration-services board.

fixes:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1073308?usp=dashboard

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1073305?usp=dashboard

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1074275?usp=dashboard

puppet is not failing anymore. Apache is, due to a cert issue, but not puppet.

Apache also fixed and running.

No more puppet errors or webserver issues here:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1074498

Dzahn added a parent task: T372804: setup gerrit2003 with gerrit service (gerrit on bookworm).Apr 24 2025, 12:36 AM