Maniphest T323798

2022-11-25: tools nfs share cleanup
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fnegri
	Nov 25 2022, 8:43 AM

Tags

Referenced Files

	F35840830: Screenshot 2022-12-06 at 12.04.14.png
	Dec 6 2022, 11:07 AM

	F35825205: Screenshot 2022-11-30 at 11.30.22.png
	Nov 30 2022, 10:33 AM

	F35818781: Screenshot 2022-11-25 at 09.52.19.png
	Nov 25 2022, 8:53 AM

Subscribers

Description

Alerting again: DISK CRITICAL - free space: /srv/tools 1074712 MB (13% inode=68%):

Dashboard: https://grafana.wikimedia.org/d/50z0i4XWz/tools-overall-nfs-storage-utilization?orgId=1&from=now-7d&to=now
Runbook: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#NFS_volume_cleanup

Previous tasks:

T206239 2018-10-04
T217993 2019-03-10
T233120 2019-09-17
T247315 2020-03-10
T261336 2020-08-26
T272247 2021-01-17
T273961 2021-02-05
T276525 2021-03-05
T284964 2021-06-15
T288276 2021-08-05
T301508 2022-02-10
T313975 2022-07-27
T318009 2022-09-16
T321839 2022-10-27

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		fnegri	T323798 2022-11-25: tools nfs share cleanup
		Resolved		wmr	T272436 wmr-bot home directory using high NFS storage
		Resolved		Andrew	T324172 'cronout' dir in tb-dev

Event Timeline

fnegri created this task.Nov 25 2022, 8:43 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 25 2022, 8:43 AM

fnegri changed the task status from Open to In Progress.Nov 25 2022, 8:43 AM

fnegri triaged this task as High priority.

fnegri added a project: cloud-services-team (Kanban).

fnegri updated the task description. (Show Details)

fnegri merged a task: T313975: 2022-07-27: tools nfs share cleanup.Nov 25 2022, 8:46 AM

fnegri added a subscriber: • nskaggs.

This graph suggests it's again wmr-bot using a lot of space, see T272436.

Screenshot 2022-11-25 at 09.52.19.png (698×2 px, 127 KB)

root@labstore1004:/srv/tools/shared/tools/home/wmr-bot# du -hs .
892G    .

fnegri claimed this task.Nov 25 2022, 8:53 AM

fnegri added a subtask: T272436: wmr-bot home directory using high NFS storage.

taavi added a project: Toolforge.Nov 25 2022, 9:03 AM

The maintainer of wmr-bot is slowly decreasing the space used by the tool (more details in T272436), and that is visible in Grafana:

Screenshot 2022-11-30 at 11.30.22.png (694×1 px, 102 KB)

fnegri moved this task from Inbox to Blocked on the cloud-services-team (Kanban) board.Nov 30 2022, 12:00 PM

wmr closed subtask T272436: wmr-bot home directory using high NFS storage as Resolved.Dec 6 2022, 10:02 AM

fnegri reopened subtask T272436: wmr-bot home directory using high NFS storage as Open.Dec 6 2022, 10:29 AM

wmr closed subtask T272436: wmr-bot home directory using high NFS storage as Resolved.Dec 6 2022, 11:04 AM

We're back under 80% occupied space.

Screenshot 2022-12-06 at 12.04.14.png (692×2 px, 125 KB)

Andrew closed subtask T324172: 'cronout' dir in tb-dev as Resolved.Jul 31 2024, 10:05 PM

taavi mentioned this in T349895: tools-nfs-2 almost out of disk space (October 2023 edition).Jan 13 2026, 4:51 PM