Page MenuHomePhabricator

2018-10-04: tools and NFS share cleanup (high usage)
Closed, ResolvedPublic

Description

In the tradition of T183920, we must use any reasonable means to reclaim disk space for the tools NFS shares.

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.8T  841G  90% /srv/tools
/dev/drbd3      5.0T  3.0T  1.8T  64% /srv/misc

Event Timeline

Bstorm created this task.Oct 4 2018, 4:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 4 2018, 4:51 PM
Bstorm triaged this task as High priority.Oct 4 2018, 4:51 PM
Bstorm renamed this task from 2019-10-04: tools and NFS share cleanup (high usage) to 2018-10-04: tools and NFS share cleanup (high usage).Oct 4 2018, 4:55 PM

Running ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_20181004.txt

Bstorm added a comment.EditedOct 4 2018, 5:45 PM

Apparently /srv/tools/shared/tools/project/.system/gridengine/spool/qmaster/messages is currently at 178G. That's kind of insane.

I may be curious about why, but any persisting condition will continue to reveal itself after I truncate it.

Bstorm added a comment.Oct 4 2018, 5:49 PM

Aaaand now the master is writing nonsense to the file.

Bstorm added a comment.Oct 4 2018, 6:17 PM

Restarting the master was a bit rocky, but it's working now.

Bstorm added a comment.Oct 4 2018, 6:18 PM

87% - next big ones are as follows

/srv/tools/shared/tools/project/.shared/dumps/20180924.json.gz 42215452 KB
/srv/tools/shared/tools/project/wikidata-todo/oneoff/statements_vs_bytes/joined.tab 42801164 KB
/srv/tools/shared/tools/project/splinetools/dumps/enwiki-20141106-pages-articles.xml 49303704 KB
/srv/tools/shared/tools/project/toolserver-home-archive/archive-2014-06-05.tar.xz 54437988 KB
/srv/tools/shared/tools/project/huggle/error.log 54733736 KB
/srv/tools/shared/tools/project/wiwosm/error.log 67078812 KB
/srv/tools/shared/tools/project/bookworm/bookworm.out 69099092 KB
/srv/tools/shared/tools/project/oar/repository_text_2014-06-13.tar.gz 78428056 KB
/srv/tools/shared/tools/home/hoo/dumpconvert.csv 105729444 KB

Truncated >100MB .err files (158GB freed):

36948936 /srv/tools/shared/tools/project/mbrt1/cron-29.err
17099564 /srv/tools/shared/tools/project/rubinbot2/refs.err
14165056 /srv/tools/shared/tools/project/khanamalumat/shaher.err
12579308 /srv/tools/shared/tools/project/robokobot/virgule.err
8157844 /srv/tools/shared/tools/project/olsibot/tung.err
5765156 /srv/tools/shared/tools/project/iabot/Workers/Worker5.err
4562784 /srv/tools/shared/tools/project/iabot/Workers/Worker1.err
4504180 /srv/tools/shared/tools/project/ping08bot/mybot.err
4146728 /srv/tools/shared/tools/project/iabot/Workers/Worker2.err
3427644 /srv/tools/shared/tools/project/iabot/Workers/Worker4.err
2770596 /srv/tools/shared/tools/project/huggle/dx.err
2668776 /srv/tools/shared/tools/project/mifterbot/cron-tools.mifterbot-commons-1.err
2325668 /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-1.err
2259016 /srv/tools/shared/tools/project/wikitanvirbot/as-welcome.err
2021652 /srv/tools/shared/tools/project/deltaquad-bots/cron-tools.deltaquad-bots-SPI.err
1813748 /srv/tools/shared/tools/project/urdubot/imla.err
1749320 /srv/tools/shared/tools/project/yifeibot/rmiw.w2.err
1650132 /srv/tools/shared/tools/project/kolbert/commonskolbert.err
1643928 /srv/tools/shared/tools/project/bub/worker1.err
1504088 /srv/tools/shared/tools/project/zumraband/commonscat.err
1410608 /srv/tools/shared/tools/project/vitalarticlesbot/vitalarticlesbot.err
1359376 /srv/tools/shared/tools/project/khanamalumat/hawai.err
1341456 /srv/tools/shared/tools/project/sammour/java.err
1095936 /srv/tools/shared/tools/project/sigma/archivebot.err
1023904 /srv/tools/shared/tools/project/fatemi/ncat.err
1001564 /srv/tools/shared/tools/project/dexbot/ghaher69.err
996432 /srv/tools/shared/tools/project/yifeibot/rmiw.w5.err
990692 /srv/tools/shared/tools/project/mix-n-match/sync.err
901948 /srv/tools/shared/tools/project/zumraband/recents.err
785564 /srv/tools/shared/tools/project/aivanalysis/afc-ores.err
783808 /srv/tools/shared/tools/home/ping08u/noreferences.err
754928 /srv/tools/shared/tools/project/csbot/csbot8.err
721236 /srv/tools/shared/tools/project/yifeibot/rmiw.w4.err
694076 /srv/tools/shared/tools/project/rezabot/wpgads.err
678424 /srv/tools/shared/tools/project/fatemi/hfnobot.err
662928 /srv/tools/shared/tools/project/wdpv/wdpv-process-and-dump.err
641516 /srv/tools/shared/tools/project/khanamalumat/usstate.err
640292 /srv/tools/shared/tools/project/yifeibot/rmiw.w3.err
633536 /srv/tools/shared/tools/home/jortola/list-replaceall.err
613052 /srv/tools/shared/tools/project/yifeibot/rmiw.w1.err
607168 /srv/tools/shared/tools/project/ashotbot/numberof.err
592980 /srv/tools/shared/tools/project/rezabot/red.err
590928 /srv/tools/shared/tools/project/himo/fixold.err
579328 /srv/tools/shared/tools/project/rezabot/radehc.err
560380 /srv/tools/shared/tools/project/hsfbot/chk.img.err
549228 /srv/tools/shared/tools/project/jarbot-ii/mcatv1.err
511764 /srv/tools/shared/tools/project/kolbert/enkolbert.err
467836 /srv/tools/shared/tools/project/invadibot/inv-es-redirects.err
462576 /srv/tools/shared/tools/project/fatemi/hfnobot2.err
440688 /srv/tools/shared/tools/project/basebot/oldlogs/basebotwlxgatherdata.err
420060 /srv/tools/shared/tools/project/mifterbot-en/cron-tools.mifterbot-en-1.err
397420 /srv/tools/shared/tools/project/yemen/news_desc.err
377604 /srv/tools/shared/tools/project/yemen/peop_desc.err
365772 /srv/tools/shared/tools/project/dimastbkbot/welcome.err
355320 /srv/tools/shared/tools/project/jarbot-ii/core/scripts/mcatv1.err
318816 /srv/tools/shared/tools/project/hsfbot/welc-id.err
311348 /srv/tools/shared/tools/project/hsfbot/welc-ms.err
310756 /srv/tools/shared/tools/project/wdvaliditycheck/cron-checkversion.err
290220 /srv/tools/shared/tools/project/rchv/java.err
289560 /srv/tools/shared/tools/project/framabot/frwiki-cleaner.err
287392 /srv/tools/shared/tools/project/liangent-php/php_dispatchRC_zhwiki.err
284080 /srv/tools/shared/tools/project/kasparbot/kb_pd_magic.err
270412 /srv/tools/shared/tools/project/jarallah-ii/addlinksv2.err
268684 /srv/tools/shared/tools/project/eranbot/enwiki.err
255176 /srv/tools/shared/tools/project/perfectbot/TemplateTranslateWrapper.err
249452 /srv/tools/shared/tools/project/hsfbot/welc-jv.err
240916 /srv/tools/shared/tools/project/himo/w_ccg.err
237696 /srv/tools/shared/tools/project/avicbot/trwprd.err
229516 /srv/tools/shared/tools/project/himo/newca.err
223896 /srv/tools/shared/tools/project/urdubot/newimla.err
221952 /srv/tools/shared/tools/project/ruwn-misc/refreshfeeds.err
205800 /srv/tools/shared/tools/project/fatemi/hfnewp.err
202096 /srv/tools/shared/tools/home/lziad/P4-bot.err
201440 /srv/tools/shared/tools/project/njsbot/sign.err
193708 /srv/tools/shared/tools/project/dimastbkbot/kds.err
183344 /srv/tools/shared/tools/project/faebot/GLAMdash.err
181736 /srv/tools/shared/tools/project/jarbot-ii/gettmlinks.err
176296 /srv/tools/shared/tools/project/mix-n-match/as_update.err
175800 /srv/tools/shared/tools/project/kenrick95bot/idwiki-welcome-2018-10-28.err
171740 /srv/tools/shared/tools/project/himo/wikidatalink.err
168224 /srv/tools/shared/tools/project/g13bot/g13_nudge.err
163408 /srv/tools/shared/tools/project/dimastbkbot/welcome_ws.err
153512 /srv/tools/shared/tools/project/fatemi/amarakhir.err
150428 /srv/tools/shared/tools/project/himo/illwd2Template.err
144848 /srv/tools/shared/tools/project/hazard-bot/d-rfbotstatus.err
142376 /srv/tools/shared/tools/project/rezabot/neededarticle.err
141364 /srv/tools/shared/tools/project/pltools/logs/cronMonthly.err
139828 /srv/tools/shared/tools/project/himo/ennew.err
138084 /srv/tools/shared/tools/project/yifeibot/comsign.err
136792 /srv/tools/shared/tools/project/aswnbot/wmfqua.err
133360 /srv/tools/shared/tools/project/wikibugs/wb2-phab.err
133308 /srv/tools/shared/tools/project/yifeibot/rmiw.commons.err
132796 /srv/tools/shared/tools/project/reports-bot/cron-update_members.err
131508 /srv/tools/shared/tools/project/media-dubiety/ircbot.err
127152 /srv/tools/shared/tools/project/yifeibot/flr.err
126308 /srv/tools/shared/tools/project/iabot/Workers/Worker3.err
117440 /srv/tools/shared/tools/project/hazard-bot/d-archivebot.err
116696 /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-3.err
112080 /srv/tools/shared/tools/project/wikisource-bot/enws-touch-all.err
111424 /srv/tools/shared/tools/project/matsubot/python3.err
109368 /srv/tools/shared/tools/project/emijrpbot/emijrpbot-newpages.err
108488 /srv/tools/shared/tools/project/deltabot/logs/cron1.err
107380 /srv/tools/shared/tools/project/hsfbot/welc-gor.err
106260 /srv/tools/shared/tools/project/eranbot/eswiki.err
105548 /srv/tools/shared/tools/project/zkbot/zk-weekly.err
104952 /srv/tools/shared/tools/project/njsbot/cleanWKT.err
102956 /srv/tools/shared/tools/project/alaabot/python2.err

Disk usage after operation:

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.8T  856G  89% /srv/tools
/dev/drbd3      5.0T  3.1T  1.7T  66% /srv/misc

Truncated >100MB error.log files (281GB freed):

67078812 /srv/tools/shared/tools/project/wiwosm/error.log
54733736 /srv/tools/shared/tools/project/huggle/error.log
27975184 /srv/tools/shared/tools/project/kmlexport/error.log
27284632 /srv/tools/shared/tools/project/magnus-toolserver/error.log
23410908 /srv/tools/shared/tools/project/wikiwatchdog/error.log
17844608 /srv/tools/shared/tools/project/request/error.log
9911176 /srv/tools/shared/tools/project/dnbtools/error.log
8759900 /srv/tools/shared/tools/project/ru_monuments/error.log
5544352 /srv/tools/shared/tools/project/denkmalliste/error.log
4719916 /srv/tools/shared/tools/project/sigma/error.log
4422592 /srv/tools/shared/tools/project/wsexport/error.log
4062748 /srv/tools/shared/tools/project/query2map/error.log
3780016 /srv/tools/shared/tools/project/refill/error.log
3267476 /srv/tools/shared/tools/project/zoomable-images/error.log
2681720 /srv/tools/shared/tools/project/quentinv57-tools/error.log
2630180 /srv/tools/shared/tools/project/drtrigonbot/error.log
2308888 /srv/tools/shared/tools/project/awb/error.log
1581460 /srv/tools/shared/tools/project/makeref/error.log
1521460 /srv/tools/shared/tools/project/catfood/error.log
1436308 /srv/tools/shared/tools/project/recoin/error.log
1350636 /srv/tools/shared/tools/project/pinyin-wiki/error.log
1329316 /srv/tools/shared/tools/project/osm4wiki/error.log
1231404 /srv/tools/shared/tools/project/wikidata-todo/error.log
1161328 /srv/tools/shared/tools/project/enwp10/error.log
1067408 /srv/tools/shared/tools/project/catscan3/error.log
1008508 /srv/tools/shared/tools/project/iluvatarbot/error.log
1000660 /srv/tools/shared/tools/project/contributionsurveyor/error.log
879960 /srv/tools/shared/tools/project/manypedia/error.log
782240 /srv/tools/shared/tools/project/joanjoc/error.log
754856 /srv/tools/shared/tools/project/admin/error.log
751616 /srv/tools/shared/tools/project/slumpartikel/error.log
701024 /srv/tools/shared/tools/project/mix-n-match/error.log
605544 /srv/tools/shared/tools/project/robin/error.log
558472 /srv/tools/shared/tools/project/ia-upload/error.log
379476 /srv/tools/shared/tools/project/hroest/error.log
372820 /srv/tools/shared/tools/project/persondata/error.log
354112 /srv/tools/shared/tools/project/reasonator/error.log
345476 /srv/tools/shared/tools/project/flickr2commons/error.log
327896 /srv/tools/shared/tools/project/kasparbot/error.log
327720 /srv/tools/shared/tools/project/aivanalysis/error.log
313792 /srv/tools/shared/tools/project/anagrimes/error.log
298744 /srv/tools/shared/tools/project/freddy2001/error.log
262356 /srv/tools/shared/tools/project/svgtranslate/error.log
257128 /srv/tools/shared/tools/project/zoomviewer/error.log
254640 /srv/tools/shared/tools/project/superyetkin/error.log
231204 /srv/tools/shared/tools/project/wikiedudashboard/error.log
222748 /srv/tools/shared/tools/project/pirsquared/error.log
221056 /srv/tools/shared/tools/project/lists/error.log
217908 /srv/tools/shared/tools/project/not-in-the-other-language/error.log
216948 /srv/tools/shared/tools/project/wmukevents/error.log
200312 /srv/tools/shared/tools/project/featured-article/error.log
178100 /srv/tools/shared/tools/project/cp/error.log
174724 /srv/tools/shared/tools/project/dow/error.log
173848 /srv/tools/shared/tools/project/locator/error.log
166436 /srv/tools/shared/tools/project/bub/error.log
159252 /srv/tools/shared/tools/project/tsreports/error.log
158284 /srv/tools/shared/tools/project/reftoolbar/error.log
152484 /srv/tools/shared/tools/project/dplbot/error.log
150576 /srv/tools/shared/tools/project/wikidata-primary-sources/error.log
141724 /srv/tools/shared/tools/project/steinsplitter/error.log
137324 /srv/tools/shared/tools/project/coord/error.log
136524 /srv/tools/shared/tools/project/wd-art/error.log
118168 /srv/tools/shared/tools/project/panoviewer/error.log
110760 /srv/tools/shared/tools/project/fiwiki-tools/error.log
106316 /srv/tools/shared/tools/project/audetools/error.log
103100 /srv/tools/shared/tools/project/templatetiger/error.log
102680 /srv/tools/shared/tools/project/wikivoyage/error.log
102516 /srv/tools/shared/tools/project/para/error.log
6700 /srv/tools/shared/tools/project/timescale/error.log

Disk usage after operation:

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.5T  1.2T  86% /srv/tools
/dev/drbd3      5.0T  3.1T  1.7T  66% /srv/misc

Current disk usage:

dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  5.6T  2.1T  74% /srv/tools
/dev/drbd3      5.0T  3.1T  1.7T  66% /srv/misc

Current disk usage:

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  5.6T  2.1T  74% /srv/tools
/dev/drbd3      5.0T  2.2T  2.6T  45% /srv/misc

For us that's pretty good. We could probably just close this one for now.

Unless we want to wait for the subtasks.

GTirloni closed this task as Resolved.Dec 1 2018, 5:40 AM
GTirloni changed the status of subtask T208466: bookworm is using 254GB of space from Open to Stalled.
GTirloni removed a subscriber: GTirloni.Dec 1 2018, 11:01 AM
Bstorm changed the status of subtask T208466: bookworm is using 254GB of space from Stalled to Open.Mar 10 2019, 11:13 PM