wdqs1006 /srv low on disk space
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Dec 24 2019, 1:56 PM

Description

24 Dec 13:38 <icinga-wm> PROBLEM - Disk space on wdqs1006 is CRITICAL: DISK CRITICAL - free space: /srv 53349 MB (3% inode=99%)

Related Objects

Mentioned In: T265868: wdqs2002 out of disk space

Event Timeline

jcrespo created this task.Dec 24 2019, 1:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 24 2019, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-24T13:58:28Z] <jynus> tune2fs -m 0 /dev/mapper/wdqs1006--vg-data T241418

The wdqs journal is 1.3T, where the usual size is ~600G. We've had similar issues before, where free space is not recovered. The only known solution at this point is copying a sane journal from another host.

Gehel added a project: Discovery-Search (Current work).Dec 24 2019, 2:02 PM

Gehel added subscribers: dcausse, • Zbyszko.

Issue started the 22 Dec at around 2:16

https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=wdqs1006&var-datasource=eqiad%20prometheus%2Fops&var-cluster=wdqs&from=1576541970193&to=1577197748564&fullscreen&panelId=12

Mentioned in SAL (#wikimedia-operations) [2019-12-24T14:13:43Z] <gehel> data reload from wdqs1008 to wdqs1006 - T241418

This is solved to me, thanks @Gehel https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&fullscreen&panelId=8&from=1577192533117&to=1577210803000

Gehel moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Dec 27 2019, 7:53 PM

Addshore moved this task from incoming to monitoring on the Wikidata board.Jan 2 2020, 9:53 AM

Gehel closed this task as Resolved.Jan 7 2020, 2:46 PM

Gehel claimed this task.

Gehel mentioned this in T265868: wdqs2002 out of disk space.Oct 19 2020, 7:48 AM

wdqs1006 /srv low on disk spaceClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

wdqs1006 /srv low on disk space
Closed, ResolvedPublic
Actions