toolforge NFS file cleanup
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Andrew
	Oct 27 2022, 4:34 PM

Description

This is alerting again.

see e.g. https://phabricator.wikimedia.org/T273961, https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#NFS_volume_cleanup

Related Objects

Mentioned In: T323798: 2022-11-25: tools nfs share cleanup
Mentioned Here: T273961: 2021-02-05: tools NFS share cleanup, v2

Event Timeline

Andrew created this task.Oct 27 2022, 4:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2022, 4:34 PM

As recommended by the runbook I have launched the following command in a screen session in labstore1004, but it's taking a long time to complete:

root@labstore1004:~# ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%k KB %p\n" | sort -h > tools_large_files_$(date +%Y%m%d).txt

Space utilization of /srv/tools is at 80.2% according to Grafana so I think we can wait until Monday before doing any cleanup.

root@labstore1004:~# tail tools_large_files_20221028.txt
30503456 KB /srv/tools/shared/tools/project/robokobot/virgule.err
30580852 KB /srv/tools/shared/tools/project/ashbot/USA.out
31437236 KB /srv/tools/shared/tools/project/freebase/freebase-rdf-latest.gz
32003152 KB /srv/tools/shared/tools/project/ifttt/uwsgi.log
32382692 KB /srv/tools/shared/tools/project/persondata/logs/person_bkl2.out
34161960 KB /srv/tools/shared/tools/project/quickstatements/tool.log
35900436 KB /srv/tools/shared/tools/project/zoomviewer/iipsrv.log
36285016 KB /srv/tools/shared/tools/project/wiki2prop/prediction_ranked_Wiki2PropDEPLOY_year2018_embedding300LG_DEPLOY.h5
37569476 KB /srv/tools/shared/tools/project/persondata/logs/wikidata_sitelinks.out
49303704 KB /srv/tools/shared/tools/project/splinetools/dumps/enwiki-20141106-pages-articles.xml

There is nothing obviously extra-large as we had for example in T273961, and the total disk size is increasing slowly (was at 80.2% on Friday, it's currently at 80.5%). I wonder if we should consider increasing the total space?

/dev/drbd4      8.0T  6.0T  1.6T  80% /srv/tools

fnegri changed the task status from Open to In Progress.Oct 31 2022, 11:48 AM

fnegri triaged this task as Medium priority.

fnegri moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Oct 31 2022, 12:12 PM

Top 10 "project" folders by size, and a comparison with the same list 1 year ago:

root@labstore1004:~# ionice -c 3 nice -19 du -hs /srv/tools/shared/tools/project/* |sort -h > /root/tools_project_size_20221031.txt

root@labstore1004:~# tail -n 10 tools_project_size_202*
==> tools_project_size_20210915.txt <==
72G     /srv/tools/shared/tools/project/digero
79G     /srv/tools/shared/tools/project/mix-n-match
84G     /srv/tools/shared/tools/project/eatchabot
111G    /srv/tools/shared/tools/project/panoviewer
111G    /srv/tools/shared/tools/project/phetools
112G    /srv/tools/shared/tools/project/templatehoard
114G    /srv/tools/shared/tools/project/wikidata-analysis
180G    /srv/tools/shared/tools/project/zoomviewer
220G    /srv/tools/shared/tools/project/glamtools
524G    /srv/tools/shared/tools/project/iabot

==> tools_project_size_20221031.txt <==
78G     /srv/tools/shared/tools/project/persondata
86G     /srv/tools/shared/tools/project/mix-n-match
109G    /srv/tools/shared/tools/project/fixsuggesterbot
112G    /srv/tools/shared/tools/project/panoviewer
114G    /srv/tools/shared/tools/project/wikidata-analysis
118G    /srv/tools/shared/tools/project/phetools
133G    /srv/tools/shared/tools/project/templatehoard
196G    /srv/tools/shared/tools/project/digero
284G    /srv/tools/shared/tools/project/zoomviewer
422G    /srv/tools/shared/tools/project/glamtools

And the top 10 "home" folders by size:

root@labstore1004:~# ionice -c 3 nice -19 du -hs /srv/tools/shared/tools/home/* |sort -h > /root/tools_home_size_20221103.txt
root@labstore1004:~# tail tools_home_size_20221103.txt
11G     /srv/tools/shared/tools/home/zhaofeng
12G     /srv/tools/shared/tools/home/davids
13G     /srv/tools/shared/tools/home/kjschiroo
13G     /srv/tools/shared/tools/home/pirsquared
14G     /srv/tools/shared/tools/home/ariutta
15G     /srv/tools/shared/tools/home/lamrin
16G     /srv/tools/shared/tools/home/cosmiclattes
22G     /srv/tools/shared/tools/home/mzmcbride
40G     /srv/tools/shared/tools/home/dispenser
189G    /srv/tools/shared/tools/home/wmr-bot

I'm about to run these in labstore1004:/srv/tools/shared/tools:

find -name '*.log' -not -newermt "Nov 1, 2021" -exec rm {} \;
find -name '*.err' -not -newermt "Nov 1, 2021" -exec rm {} \;
find -name '*.log'  -size +1G -exec truncate --size=1G {} \;
find -name '*.err'  -size +1G -exec truncate --size=1G {} \;

Used disk space has decreased from 80% to 75%:

/dev/drbd4      8.0T  5.6T  2.0T  75% /srv/tools

Resolving for now.

fnegri mentioned this in T323798: 2022-11-25: tools nfs share cleanup.Nov 25 2022, 8:43 AM

toolforge NFS file cleanupClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

toolforge NFS file cleanup
Closed, ResolvedPublic
Actions