2020-08-26: tools NFS share cleanup
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Bstorm
	Aug 26 2020, 5:09 PM

Description

Once again, in the tradition of T156982: Cleanup tools nfs share on labstore1004/5 and T247315: 2020-03-10: tools and misc nfs share cleanup, we must clean up the NFS to prevent collapse

This time, just /srv/tools

/dev/drbd4      8.0T  6.9T  745G  91% /srv/tools
/dev/drbd3      5.0T  1.9T  2.9T  40% /srv/misc

Projects using the most storage in grafana

Related Objects
Search...

Status	Assigned	Task
Resolved	• Bstorm	T261335 Fix alerting for disk space on the NFS servers
Resolved	Andrew	T261336 2020-08-26: tools NFS share cleanup
Resolved	• nskaggs	T248188 Zoomviewer has ~450,000 files in NFS home directory
Resolved	• Bstorm	T261338 Remove PAWS data in Tools volume
Resolved	Wurgl	T261339 persondata tool is using a large amount of NFS space -- please try to clean it up
Resolved	bennofs	T247449 wdumps custom generated dumps storage space

Event Timeline

• Bstorm removed • Bstorm as the assignee of this task.Aug 26 2020, 5:09 PM

• Bstorm triaged this task as High priority.

• Bstorm created this task.

• Bstorm moved this task from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.

Mentioned in SAL (#wikimedia-cloud) [2020-08-26T17:12:48Z] <bstorm> Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%k KB %p\n" > tools_large_files_20200826.txt' on labstore1004 T261336

• Bstorm updated the task description. (Show Details)Aug 26 2020, 5:15 PM

• Bstorm added a subtask: T248188: Zoomviewer has ~450,000 files in NFS home directory .

• Bstorm added a subtask: T247449: wdumps custom generated dumps storage space.Aug 26 2020, 5:25 PM

cat tools_large_files_20200826.txt | sort -h > tools_large_files-sorted_20200826.txt

Top individual files are:

49303704 KB /srv/tools/shared/tools/project/splinetools/dumps/enwiki-20141106-pages-articles.xml
51288456 KB /srv/tools/shared/tools/project/persondata/dumps/dnb-all_lds.rdf
63967176 KB /srv/tools/shared/tools/project/magnus-toolserver/error.log
64778744 KB /srv/tools/shared/tools/project/wikidata-analysis/public_html_tmp/dumpfiles/json-20191125/20191125.json.gz
77725816 KB /srv/tools/shared/tools/project/persondata/logs/person_bkl2.out
85264568 KB /srv/tools/shared/tools/project/.shared/dumps/20200720.json.gz
85736004 KB /srv/tools/shared/tools/project/.shared/dumps/20200803.json.gz
86201908 KB /srv/tools/shared/tools/project/.shared/dumps/20200817.json.gz
101160884 KB /srv/tools/shared/tools/project/robokobot/virgule.err
410869488 KB /srv/tools/shared/tools/project/persondata/logs/wikidata_sitelinks.out

• Bstorm closed subtask T261338: Remove PAWS data in Tools volume as Resolved.Aug 26 2020, 5:46 PM

Nintendofan885 subscribed.Aug 26 2020, 6:29 PM

Truncated /srv/tools/shared/tools/project/robokobot/virgule.err

• nskaggs subscribed.Aug 27 2020, 3:50 PM