Maniphest T399261

Widespread instances down in project deployment-prep
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	wmcs-alerts
	Jul 11 2025, 7:19 AM

Tags

Referenced Files

	F63886612: Screenshot 2025-07-11 at 11.11.26.png
	Jul 11 2025, 9:12 AM

Subscribers

Description

Common information

summary: Widespread instances down in project deployment-prep

alertname: WidespreadInstanceDown
project: deployment-prep
severity: critical

Firing alerts

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Andrew	T399858 Cloud Ceph misbehaving on Debian Bookworm
		Resolved		taavi	T399870 [k8s,infra,o11y] Add paging alert when many tools are unreachable
		Resolved		Andrew	T399281 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures
		Resolved		hashar	T399283 Remove reliance on EXECUTOR_NUMBER environment variable in CI
		Duplicate		None	T399303 Beta cluster down 2025-07-11
		Resolved		fnegri	T399261 Widespread instances down in project deployment-prep

Event Timeline

wmcs-alerts created this task.Jul 11 2025, 7:19 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 11 2025, 7:19 AM

wmcs-alerts updated the task description. (Show Details)Jul 11 2025, 7:24 AM

wmcs-alerts updated the task description. (Show Details)Jul 11 2025, 8:18 AM

wmcs-alerts updated the task description. (Show Details)Jul 11 2025, 8:23 AM

Possibly related to T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures

This is now resolved, but there were two separate spikes of unreachable instances:

Screenshot 2025-07-11 at 11.11.26.png (1×2 px, 293 KB)

Lucas_Werkmeister_WMDE added a parent task: T399303: Beta cluster down 2025-07-11.Jul 11 2025, 1:10 PM