ws-export is down (2023-10-03)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MarcoAurelio
	Oct 3 2023, 8:57 PM

Description

https://ws-export.wmcloud.org is down and errors (error page footer mentions: proxy-03.project-proxy.eqiad1.wikimedia.cloud).

Grafana mentions that wsexport-prod01 is down and two puppet not running/stale alerts for the ia-upload-prod instance for this project.

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

~~Unfortunately I do not have access to the WMCS project, but triaging on our kanban for attention~~

Mentioned in SAL (#wikimedia-cloud) [2023-10-03T21:59:24Z] <TheresNoTime> Rebooted wsexport-prod01, volume did not mount, ran sudo mount -a T348068

wsexport-prod01 had OOM'd
Did a reboot via horizon
On reboot complete, noted error creating cache directory
Using https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource/Wikisource_Export, realised that the volume had not mounted (despite being in /etc/fstab)
Did sudo mount -a
https://ws-export.wmcloud.org appears to be working as expected (@MarcoAurelio, please confirm)

In T348068#9222807, @TheresNoTime wrote:

https://ws-export.wmcloud.org appears to be working as expected (@MarcoAurelio, please confirm)