SystemdUnitFailed - gitlab - backup-restore.service / zuul-scheduler on zuul1001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Apr 10 2026, 1:30 AM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
prometheus: ops
severity: critical
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: backup-restore.service on gitlab2002:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: backup-restore.service on gitlab2002:9100
alertname: SystemdUnitFailed
instance: gitlab2002:9100
name: backup-restore.service
prometheus: ops
severity: critical
site: codfw
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: zuul-scheduler.service on zuul1001:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: zuul-scheduler.service on zuul1001:9100
alertname: SystemdUnitFailed
instance: zuul1001:9100
name: zuul-scheduler.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
zuul: make gerrit ssh key configurable in Hiera and add it	operations/puppet	production	+14 -2
add fake keys for new zuul to connect to gerrit	labs/private	master	+2 -0
zuul: mount /var/ssh/zuul for zuul-scheduler	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
					Restricted Task
		Resolved		Dzahn	T422895 SystemdUnitFailed - gitlab - backup-restore.service / zuul-scheduler on zuul1001

Event Timeline

phaultfinder created this task.Apr 10 2026, 1:30 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 10 2026, 1:30 AM

phaultfinder updated the task description. (Show Details)Apr 10 2026, 2:05 AM

phaultfinder updated the task description. (Show Details)Apr 10 2026, 7:00 AM

phaultfinder updated the task description. (Show Details)Apr 10 2026, 11:00 AM

Dzahn renamed this task from SystemdUnitFailed to SystemdUnitFailed - gitlab - backup-restore.service.Apr 10 2026, 2:50 PM

Dzahn merged a task: T422948: SystemdUnitFailed.Apr 10 2026, 4:09 PM

Dzahn added a parent task: Restricted Task.

The backup-restore on gitlab is the expected issue after the linked security upgrade.

zuul-scheduler.service und zuul1001 taking a look now.

The original issue was FileNotFoundError: [Errno 2] No such file or directory: '/var/ssh/zuul'

That was fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260847

But we are not mounting the path yet for the docker container.

Dzahn renamed this task from SystemdUnitFailed - gitlab - backup-restore.service to SystemdUnitFailed - gitlab - backup-restore.service / zuul-scheduler on zuul1001.Apr 10 2026, 4:16 PM

Dzahn claimed this task.

Change #1270103 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: mount /var/ssh/zuul for zuul-scheduler

https://gerrit.wikimedia.org/r/1270103

gerritbot added a project: Patch-For-Review.Apr 10 2026, 11:33 PM

Change #1270103 merged by Dzahn:

[operations/puppet@production] zuul: mount /var/ssh/zuul for zuul-scheduler

https://gerrit.wikimedia.org/r/1270103

Maintenance_bot removed a project: Patch-For-Review.Apr 11 2026, 12:30 AM

LSobanski triaged this task as Medium priority.Apr 13 2026, 3:54 PM

LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.

Created a new ed25519 key pair for new zuul to connect to Gerrit (in the future).

[Ops] [puppet-private] (3334cf48f) (dzahn) add new ed25519 keypair for new zuul to connect to gerrit (T395938)

It lives under secrets/gerrit/zuul_gerrit_ed25519(.pub). Has NOT been added on the Gerrit side yet.

Change #1270577 had a related patch set uploaded (by Dzahn; author: Dzahn):

[labs/private@master] add fake keys for new zuul to connect to gerrit

https://gerrit.wikimedia.org/r/1270577

Change #1270577 merged by Dzahn:

[labs/private@master] add fake keys for new zuul to connect to gerrit

https://gerrit.wikimedia.org/r/1270577

Change #1270580 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: make gerrit ssh key configurable in Hiera and add it

https://gerrit.wikimedia.org/r/1270580

Dzahn merged a task: T423315: SystemdUnitFailed - zuul-scheduler.Apr 14 2026, 5:34 PM

Change #1270580 merged by Dzahn:

[operations/puppet@production] zuul: make gerrit ssh key configurable in Hiera and add it

https://gerrit.wikimedia.org/r/1270580

Maintenance_bot removed a project: Patch-For-Review.Apr 17 2026, 7:30 PM

The previous error message is gone. The new SSH key exists. It simply has not been added on the Gerrit side, as intended. Therefore:

Apr 17 19:36:52 zuul1001 docker[2781962]: 2026-04-17 19:36:52,181 ERROR zuul.GerritConnection.ssh:   paramiko.ssh_exception.AuthenticationException: Authentication failed.

Dzahn closed this task as Resolved.Apr 17 2026, 7:37 PM

SystemdUnitFailed - gitlab - backup-restore.service / zuul-scheduler on zuul1001Closed, ResolvedPublicActions