2021-06-15: Tools NFS share cleanup
Closed, DuplicatePublic
Actions

Assigned To

Authored By

	• Bstorm
	Jun 14 2021, 11:51 PM

Description

In the tradition of T276525

We are back to do more share cleanup. This time, I'm making a task to retrofit quotas on the NFS now that other problems are largely fixed. This procedure is happening too often.

Icinga has alerted, and I must answer.

Current df output:
/dev/drbd4 8.0T 6.4T 1.3T 85% /srv/tools

Docs https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#NFS_volume_cleanup
Most useful dashboard https://grafana.wikimedia.org/d/50z0i4XWz/tools-overall-nfs-storage-utilization?orgId=1

Details

Other Assignee: • nskaggs

	Subject	Repo	Branch	Lines +/-
	nfs prometheus: change to strings for dir sizes	operations/puppet	production	+5 -4

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Duplicate	• Bstorm	T284964 2021-06-15: Tools NFS share cleanup
Resolved	Legoktm	T284965 coverme tool using a lot of space on NFS, please clean up
Open	None	T284967 Implement filesystem quotas on NFS volumes that are multi-tenant
Open	None	T284968 eatchabot using a lot of NFS storage
Resolved	bennofs	T247449 wdumps custom generated dumps storage space
Resolved	Samwilson	T284528 Move IA Upload from Toolforge to VPS
Resolved	Magnus	T285017 glamtools using increasing amounts of space on NFS
Resolved	tstarling	T285018 zoomviewer taking up a lot of NFS space -- please clean up
Resolved	Prolineserver	T285019 videoconvert taking up a lot of space on Toolforge nfs, please clean up if possible
Resolved	zhuyifei1999	T285020 /data/project/yifeibot is getting large (129G), please clean up
Resolved	bd808	T285179 meetbot has 500G of log files

Event Timeline

• Bstorm created this task.Jun 14 2021, 11:51 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 14 2021, 11:51 PM

Adding @nskaggs in case he wants to pursue some of the subtasks as the clinic duty.

Running the large-files find in a screen session.

• Bstorm updated the task description. (Show Details)Jun 14 2021, 11:59 PM

• Bstorm added a subtask: T284966: cluebotng using high storage on NFS.Jun 15 2021, 12:16 AM

Mentioned in SAL (#wikimedia-cloud) [2021-06-15T01:18:13Z] <bstorm> running a modified version of the prometheus dir size cron in screen T284964

Mentioned in SAL (#wikimedia-cloud) [2021-06-15T15:54:43Z] <bstorm> truncated 42GB virgule.err file T284964

Mentioned in SAL (#wikimedia-cloud) [2021-06-15T16:08:08Z] <bstorm> truncated 28GB person_bkl2.out T284964

Mentioned in SAL (#wikimedia-cloud) [2021-06-15T16:31:20Z] <bstorm> truncated 26GB error.log T284964

Change 699973 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] nfs prometheus: change to strings for dir sizes

https://gerrit.wikimedia.org/r/699973

gerritbot added a project: Patch-For-Review.Jun 15 2021, 9:04 PM

I ran some of our prometheus stuff in a modified fashion to find out what is actually the biggest users because it seems like there's no way the ones identified so far are taking all this up. Unfortunately, that was correct. Here are the largest projects:
templatehoard 93.52 GB
phetools 101.71 GB
panoviewer 110.16 GB
wikidata-analysis 113.03 GB
yifeibot 128.91 GB
videoconvert 140.03 GB
zoomviewer 174.70 GB
glamtools 182.25 GB
ia-upload 290.00 GB
wdumps 310.95 GB