Maniphest T399303

Beta cluster down 2025-07-11
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	Lucas_Werkmeister_WMDE
	Jul 11 2025, 1:09 PM

Tags

Referenced Files

None

Subscribers

Lucas_Werkmeister_WMDE

Description

The beta cluster has been unreachable (timing out) for the better part of today, as far as I’m aware (under both its old and new domain names). The Ceph issues at T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures are a likely culprit according to T399281#10995431.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Andrew	T399858 Cloud Ceph misbehaving on Debian Bookworm
		Resolved		taavi	T399870 [k8s,infra,o11y] Add paging alert when many tools are unreachable
		Resolved		Andrew	T399281 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures
		Resolved		hashar	T399283 Remove reliance on EXECUTOR_NUMBER environment variable in CI
		Duplicate		None	T399303 Beta cluster down 2025-07-11
		Resolved		fnegri	T399261 Widespread instances down in project deployment-prep
		Duplicate		None	T399297 Widespread instances down in project deployment-prep

Event Timeline

Lucas_Werkmeister_WMDE created this task.Jul 11 2025, 1:09 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 11 2025, 1:09 PM

Lucas_Werkmeister_WMDE added subtasks: T399216: No Puppet resources found on instance deployment-urldownloader04 on project deployment-prep, T399261: Widespread instances down in project deployment-prep, T399297: Widespread instances down in project deployment-prep.Jul 11 2025, 1:10 PM

The beta-code-update-eqiad, beta-scap-sync-world and beta-update-databases-eqiad CI jobs have been green throughout FWIW.

Lucas_Werkmeister_WMDE mentioned this in T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures.Jul 11 2025, 1:26 PM

Novem_Linguae subscribed.Jul 11 2025, 1:28 PM

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Jul 11 2025, 1:34 PM

Lucas_Werkmeister_WMDE added a parent task: T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures.

hashar mentioned this in T399283: Remove reliance on EXECUTOR_NUMBER environment variable in CI.Jul 11 2025, 2:39 PM

hashar added a parent task: T399283: Remove reliance on EXECUTOR_NUMBER environment variable in CI.

As noted: https://phabricator.wikimedia.org/T289318#10995040

• bd808 closed this task as a duplicate of T399329: 2025-07-11 traffic overload.Jul 11 2025, 9:10 PM

• bd808 removed a subtask: T399216: No Puppet resources found on instance deployment-urldownloader04 on project deployment-prep.Jul 11 2025, 9:27 PM