SystemdUnitFailed - people1005 - envoyproxy
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Sep 2 2025, 7:41 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

alertname: SystemdUnitFailed
instance: people1005:9100
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services

Firing alerts

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: envoyproxy.service on people1005:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: envoyproxy.service on people1005:9100
alertname: SystemdUnitFailed
instance: people1005:9100
name: envoyproxy.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

dashboard: https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status
description: wmf_auto_restart_envoyproxy.service on people1005:9100
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
summary: wmf_auto_restart_envoyproxy.service on people1005:9100
alertname: SystemdUnitFailed
instance: people1005:9100
name: wmf_auto_restart_envoyproxy.service
prometheus: ops
severity: critical
site: eqiad
source: prometheus
team: collaboration-services
Source

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	site: add peopleweb role to new peopleweb hosts again	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	RLazarus	T380211 Upgrade Envoy to >= 1.24
Resolved	RLazarus	T402584 Upgrade Envoy to v1.26.8 and drop buster
Resolved	Dzahn	T402596 upgrade people servers to trixie
Resolved	Dzahn	T403526 SystemdUnitFailed - people1005 - envoyproxy

Event Timeline

phaultfinder created this task.Sep 2 2025, 7:41 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2025, 7:41 PM

LSobanski renamed this task from SystemdUnitFailed to SystemdUnitFailed - people1005 - envoyproxy.Sep 3 2025, 11:09 AM

LSobanski assigned this task to Dzahn.

Change #1184553 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: add peopleweb role to new peopleweb hosts again

https://gerrit.wikimedia.org/r/1184553

gerritbot added a project: Patch-For-Review.Sep 3 2025, 3:17 PM

Change #1184553 merged by Dzahn:

[operations/puppet@production] site: add peopleweb role to new peopleweb hosts again

https://gerrit.wikimedia.org/r/1184553

"wmf_auto_restart_envoyproxy.service" failing is not a surprise when envoyproxy just did not exist.. so there was nothing to restart. now there is again.

but actual envoyproxy itself failing to start IS surprising and another thing to debug now .. it should have just worked

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2025, 3:30 PM

It turned out the /etc/envoy/envoy.yaml config file was simply empty.

I had some vague memory this happened to me in the past due to some race condition.

So then just did a rm -rf /etc/envoy and ran puppet to let it attempt to recreate it all from scratch.. and it did.. and then the file had content and envoy started.

After this there was another issue with apache restart on people2004 with an error that the CAS LoginUrl was not configured.. but the values WERE in config files and somehow after a couple restarts the issue was gone.-

Dzahn added a parent task: T402596: upgrade people servers to trixie.Sep 3 2025, 4:09 PM

SystemdUnitFailed - people1005 - envoyproxyClosed, ResolvedPublicActions