Maniphest T438786

SystemdUnitFailed - zuul1005
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Mon, Sep 21, 7:58 PM

Project Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: zuul-executor.service on zuul1005:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: zuul-executor.service on zuul1005:9100

alertname: SystemdUnitFailed
instance: zuul1005:9100
name: zuul-executor.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: zuul-executor.service on zuul1005:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: zuul-executor.service on zuul1005:9100
alertname: SystemdUnitFailed
instance: zuul1005:9100
name: zuul-executor.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

Related Objects

Mentioned Here: T427353: Repurpose ganeti102[3456] for Zuul migration
T435499: Move the majority of the Registry's docker image prefixes to a new s3 bucket

Event Timeline

phaultfinder created this task.Mon, Sep 21, 7:58 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Sep 21, 7:58 PM

comes from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1319547

LSobanski merged a task: T438798: SystemdUnitFailed.Tue, Sep 22, 9:07 AM

LSobanski moved this task from Incoming to Work in Progress on the Collaboration-Services board.

LSobanski triaged this task as Medium priority.Tue, Sep 22, 11:18 AM

Alert silenced for 24 hours with 31530d81-c143-46f5-bb13-fc8a2ef985d4

Sep 22 19:23:33 zuul1005 docker[1565886]: Unable to find image 'docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-executor:wmf-14.2.0-1@sha256:c86be669652ebb0b00af6aa3e4e8e6bd269fdc>

Looking at https://docker-registry.wikimedia.org/ I do not see any releng/zuul/zuul at all?? Asked in Slack.

Dzahn merged a task: T438959: SystemdUnitFailed - zuul1005 - zuul-executor.Wed, Sep 23, 3:17 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-23T15:35:16Z] <mutante> zuul1005 - reimage - should not have had nftables on it before T438786

The cause was that docker registry moved to Swift.

It was fixed by elukey over in T435499#12353517

Still have to reimage this host but that is happening in T427353.

Dzahn closed this task as Resolved.Thu, Sep 24, 5:37 PM