Page MenuHomePhabricator
Create Task
Maniphest T206239

2018-10-04: tools and NFS share cleanup (high usage)
Closed, ResolvedPublic
Actions

Description

In the tradition of T183920, we must use any reasonable means to reclaim disk space for the tools NFS shares.

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.8T  841G  90% /srv/tools
/dev/drbd3      5.0T  3.0T  1.8T  64% /srv/misc

Related Objects
Search...

Event Timeline

Bstorm created this task.Oct 4 2018, 4:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 4 2018, 4:51 PM
Bstorm triaged this task as High priority.Oct 4 2018, 4:51 PM
Bstorm renamed this task from 2019-10-04: tools and NFS share cleanup (high usage) to 2018-10-04: tools and NFS share cleanup (high usage).Oct 4 2018, 4:55 PM

Running ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_20181004.txt

Bstorm added a comment.EditedOct 4 2018, 5:45 PM

Apparently /srv/tools/shared/tools/project/.system/gridengine/spool/qmaster/messages is currently at 178G. That's kind of insane.

I may be curious about why, but any persisting condition will continue to reveal itself after I truncate it.

Bstorm added a comment.Oct 4 2018, 5:49 PM

Aaaand now the master is writing nonsense to the file.

Bstorm added a comment.Oct 4 2018, 6:17 PM

Restarting the master was a bit rocky, but it's working now.

Bstorm added a comment.Oct 4 2018, 6:18 PM

87% - next big ones are as follows

/srv/tools/shared/tools/project/.shared/dumps/20180924.json.gz 42215452 KB
/srv/tools/shared/tools/project/wikidata-todo/oneoff/statements_vs_bytes/joined.tab 42801164 KB
/srv/tools/shared/tools/project/splinetools/dumps/enwiki-20141106-pages-articles.xml 49303704 KB
/srv/tools/shared/tools/project/toolserver-home-archive/archive-2014-06-05.tar.xz 54437988 KB
/srv/tools/shared/tools/project/huggle/error.log 54733736 KB
/srv/tools/shared/tools/project/wiwosm/error.log 67078812 KB
/srv/tools/shared/tools/project/bookworm/bookworm.out 69099092 KB
/srv/tools/shared/tools/project/oar/repository_text_2014-06-13.tar.gz 78428056 KB
/srv/tools/shared/tools/home/hoo/dumpconvert.csv 105729444 KB
Bstorm moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.Oct 18 2018, 3:59 PM
• GTirloni added a comment.Oct 31 2018, 6:01 PM

Truncated >100MB .err files (158GB freed):

36948936 /srv/tools/shared/tools/project/mbrt1/cron-29.err
17099564 /srv/tools/shared/tools/project/rubinbot2/refs.err
14165056 /srv/tools/shared/tools/project/khanamalumat/shaher.err
12579308 /srv/tools/shared/tools/project/robokobot/virgule.err
8157844 /srv/tools/shared/tools/project/olsibot/tung.err
5765156 /srv/tools/shared/tools/project/iabot/Workers/Worker5.err
4562784 /srv/tools/shared/tools/project/iabot/Workers/Worker1.err
4504180 /srv/tools/shared/tools/project/ping08bot/mybot.err
4146728 /srv/tools/shared/tools/project/iabot/Workers/Worker2.err
3427644 /srv/tools/shared/tools/project/iabot/Workers/Worker4.err
2770596 /srv/tools/shared/tools/project/huggle/dx.err
2668776 /srv/tools/shared/tools/project/mifterbot/cron-tools.mifterbot-commons-1.err
2325668 /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-1.err
2259016 /srv/tools/shared/tools/project/wikitanvirbot/as-welcome.err
2021652 /srv/tools/shared/tools/project/deltaquad-bots/cron-tools.deltaquad-bots-SPI.err
1813748 /srv/tools/shared/tools/project/urdubot/imla.err
1749320 /srv/tools/shared/tools/project/yifeibot/rmiw.w2.err
1650132 /srv/tools/shared/tools/project/kolbert/commonskolbert.err
1643928 /srv/tools/shared/tools/project/bub/worker1.err
1504088 /srv/tools/shared/tools/project/zumraband/commonscat.err
1410608 /srv/tools/shared/tools/project/vitalarticlesbot/vitalarticlesbot.err
1359376 /srv/tools/shared/tools/project/khanamalumat/hawai.err
1341456 /srv/tools/shared/tools/project/sammour/java.err
1095936 /srv/tools/shared/tools/project/sigma/archivebot.err
1023904 /srv/tools/shared/tools/project/fatemi/ncat.err
1001564 /srv/tools/shared/tools/project/dexbot/ghaher69.err
996432 /srv/tools/shared/tools/project/yifeibot/rmiw.w5.err
990692 /srv/tools/shared/tools/project/mix-n-match/sync.err
901948 /srv/tools/shared/tools/project/zumraband/recents.err
785564 /srv/tools/shared/tools/project/aivanalysis/afc-ores.err
783808 /srv/tools/shared/tools/home/ping08u/noreferences.err
754928 /srv/tools/shared/tools/project/csbot/csbot8.err
721236 /srv/tools/shared/tools/project/yifeibot/rmiw.w4.err
694076 /srv/tools/shared/tools/project/rezabot/wpgads.err
678424 /srv/tools/shared/tools/project/fatemi/hfnobot.err
662928 /srv/tools/shared/tools/project/wdpv/wdpv-process-and-dump.err
641516 /srv/tools/shared/tools/project/khanamalumat/usstate.err
640292 /srv/tools/shared/tools/project/yifeibot/rmiw.w3.err
633536 /srv/tools/shared/tools/home/jortola/list-replaceall.err
613052 /srv/tools/shared/tools/project/yifeibot/rmiw.w1.err
607168 /srv/tools/shared/tools/project/ashotbot/numberof.err
592980 /srv/tools/shared/tools/project/rezabot/red.err
590928 /srv/tools/shared/tools/project/himo/fixold.err
579328 /srv/tools/shared/tools/project/rezabot/radehc.err
560380 /srv/tools/shared/tools/project/hsfbot/chk.img.err
549228 /srv/tools/shared/tools/project/jarbot-ii/mcatv1.err
511764 /srv/tools/shared/tools/project/kolbert/enkolbert.err
467836 /srv/tools/shared/tools/project/invadibot/inv-es-redirects.err
462576 /srv/tools/shared/tools/project/fatemi/hfnobot2.err
440688 /srv/tools/shared/tools/project/basebot/oldlogs/basebotwlxgatherdata.err
420060 /srv/tools/shared/tools/project/mifterbot-en/cron-tools.mifterbot-en-1.err
397420 /srv/tools/shared/tools/project/yemen/news_desc.err
377604 /srv/tools/shared/tools/project/yemen/peop_desc.err
365772 /srv/tools/shared/tools/project/dimastbkbot/welcome.err
355320 /srv/tools/shared/tools/project/jarbot-ii/core/scripts/mcatv1.err
318816 /srv/tools/shared/tools/project/hsfbot/welc-id.err
311348 /srv/tools/shared/tools/project/hsfbot/welc-ms.err
310756 /srv/tools/shared/tools/project/wdvaliditycheck/cron-checkversion.err
290220 /srv/tools/shared/tools/project/rchv/java.err
289560 /srv/tools/shared/tools/project/framabot/frwiki-cleaner.err
287392 /srv/tools/shared/tools/project/liangent-php/php_dispatchRC_zhwiki.err
284080 /srv/tools/shared/tools/project/kasparbot/kb_pd_magic.err
270412 /srv/tools/shared/tools/project/jarallah-ii/addlinksv2.err
268684 /srv/tools/shared/tools/project/eranbot/enwiki.err
255176 /srv/tools/shared/tools/project/perfectbot/TemplateTranslateWrapper.err
249452 /srv/tools/shared/tools/project/hsfbot/welc-jv.err
240916 /srv/tools/shared/tools/project/himo/w_ccg.err
237696 /srv/tools/shared/tools/project/avicbot/trwprd.err
229516 /srv/tools/shared/tools/project/himo/newca.err
223896 /srv/tools/shared/tools/project/urdubot/newimla.err
221952 /srv/tools/shared/tools/project/ruwn-misc/refreshfeeds.err
205800 /srv/tools/shared/tools/project/fatemi/hfnewp.err
202096 /srv/tools/shared/tools/home/lziad/P4-bot.err
201440 /srv/tools/shared/tools/project/njsbot/sign.err
193708 /srv/tools/shared/tools/project/dimastbkbot/kds.err
183344 /srv/tools/shared/tools/project/faebot/GLAMdash.err
181736 /srv/tools/shared/tools/project/jarbot-ii/gettmlinks.err
176296 /srv/tools/shared/tools/project/mix-n-match/as_update.err
175800 /srv/tools/shared/tools/project/kenrick95bot/idwiki-welcome-2018-10-28.err
171740 /srv/tools/shared/tools/project/himo/wikidatalink.err
168224 /srv/tools/shared/tools/project/g13bot/g13_nudge.err
163408 /srv/tools/shared/tools/project/dimastbkbot/welcome_ws.err
153512 /srv/tools/shared/tools/project/fatemi/amarakhir.err
150428 /srv/tools/shared/tools/project/himo/illwd2Template.err
144848 /srv/tools/shared/tools/project/hazard-bot/d-rfbotstatus.err
142376 /srv/tools/shared/tools/project/rezabot/neededarticle.err
141364 /srv/tools/shared/tools/project/pltools/logs/cronMonthly.err
139828 /srv/tools/shared/tools/project/himo/ennew.err
138084 /srv/tools/shared/tools/project/yifeibot/comsign.err
136792 /srv/tools/shared/tools/project/aswnbot/wmfqua.err
133360 /srv/tools/shared/tools/project/wikibugs/wb2-phab.err
133308 /srv/tools/shared/tools/project/yifeibot/rmiw.commons.err
132796 /srv/tools/shared/tools/project/reports-bot/cron-update_members.err
131508 /srv/tools/shared/tools/project/media-dubiety/ircbot.err
127152 /srv/tools/shared/tools/project/yifeibot/flr.err
126308 /srv/tools/shared/tools/project/iabot/Workers/Worker3.err
117440 /srv/tools/shared/tools/project/hazard-bot/d-archivebot.err
116696 /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-3.err
112080 /srv/tools/shared/tools/project/wikisource-bot/enws-touch-all.err
111424 /srv/tools/shared/tools/project/matsubot/python3.err
109368 /srv/tools/shared/tools/project/emijrpbot/emijrpbot-newpages.err
108488 /srv/tools/shared/tools/project/deltabot/logs/cron1.err
107380 /srv/tools/shared/tools/project/hsfbot/welc-gor.err
106260 /srv/tools/shared/tools/project/eranbot/eswiki.err
105548 /srv/tools/shared/tools/project/zkbot/zk-weekly.err
104952 /srv/tools/shared/tools/project/njsbot/cleanWKT.err
102956 /srv/tools/shared/tools/project/alaabot/python2.err

Disk usage after operation:

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.8T  856G  89% /srv/tools
/dev/drbd3      5.0T  3.1T  1.7T  66% /srv/misc

Truncated >100MB error.log files (281GB freed):

67078812 /srv/tools/shared/tools/project/wiwosm/error.log
54733736 /srv/tools/shared/tools/project/huggle/error.log
27975184 /srv/tools/shared/tools/project/kmlexport/error.log
27284632 /srv/tools/shared/tools/project/magnus-toolserver/error.log
23410908 /srv/tools/shared/tools/project/wikiwatchdog/error.log
17844608 /srv/tools/shared/tools/project/request/error.log
9911176 /srv/tools/shared/tools/project/dnbtools/error.log
8759900 /srv/tools/shared/tools/project/ru_monuments/error.log
5544352 /srv/tools/shared/tools/project/denkmalliste/error.log
4719916 /srv/tools/shared/tools/project/sigma/error.log
4422592 /srv/tools/shared/tools/project/wsexport/error.log
4062748 /srv/tools/shared/tools/project/query2map/error.log
3780016 /srv/tools/shared/tools/project/refill/error.log
3267476 /srv/tools/shared/tools/project/zoomable-images/error.log
2681720 /srv/tools/shared/tools/project/quentinv57-tools/error.log
2630180 /srv/tools/shared/tools/project/drtrigonbot/error.log
2308888 /srv/tools/shared/tools/project/awb/error.log
1581460 /srv/tools/shared/tools/project/makeref/error.log
1521460 /srv/tools/shared/tools/project/catfood/error.log
1436308 /srv/tools/shared/tools/project/recoin/error.log
1350636 /srv/tools/shared/tools/project/pinyin-wiki/error.log
1329316 /srv/tools/shared/tools/project/osm4wiki/error.log
1231404 /srv/tools/shared/tools/project/wikidata-todo/error.log
1161328 /srv/tools/shared/tools/project/enwp10/error.log
1067408 /srv/tools/shared/tools/project/catscan3/error.log
1008508 /srv/tools/shared/tools/project/iluvatarbot/error.log
1000660 /srv/tools/shared/tools/project/contributionsurveyor/error.log
879960 /srv/tools/shared/tools/project/manypedia/error.log
782240 /srv/tools/shared/tools/project/joanjoc/error.log
754856 /srv/tools/shared/tools/project/admin/error.log
751616 /srv/tools/shared/tools/project/slumpartikel/error.log
701024 /srv/tools/shared/tools/project/mix-n-match/error.log
605544 /srv/tools/shared/tools/project/robin/error.log
558472 /srv/tools/shared/tools/project/ia-upload/error.log
379476 /srv/tools/shared/tools/project/hroest/error.log
372820 /srv/tools/shared/tools/project/persondata/error.log
354112 /srv/tools/shared/tools/project/reasonator/error.log
345476 /srv/tools/shared/tools/project/flickr2commons/error.log
327896 /srv/tools/shared/tools/project/kasparbot/error.log
327720 /srv/tools/shared/tools/project/aivanalysis/error.log
313792 /srv/tools/shared/tools/project/anagrimes/error.log
298744 /srv/tools/shared/tools/project/freddy2001/error.log
262356 /srv/tools/shared/tools/project/svgtranslate/error.log
257128 /srv/tools/shared/tools/project/zoomviewer/error.log
254640 /srv/tools/shared/tools/project/superyetkin/error.log
231204 /srv/tools/shared/tools/project/wikiedudashboard/error.log
222748 /srv/tools/shared/tools/project/pirsquared/error.log
221056 /srv/tools/shared/tools/project/lists/error.log
217908 /srv/tools/shared/tools/project/not-in-the-other-language/error.log
216948 /srv/tools/shared/tools/project/wmukevents/error.log
200312 /srv/tools/shared/tools/project/featured-article/error.log
178100 /srv/tools/shared/tools/project/cp/error.log
174724 /srv/tools/shared/tools/project/dow/error.log
173848 /srv/tools/shared/tools/project/locator/error.log
166436 /srv/tools/shared/tools/project/bub/error.log
159252 /srv/tools/shared/tools/project/tsreports/error.log
158284 /srv/tools/shared/tools/project/reftoolbar/error.log
152484 /srv/tools/shared/tools/project/dplbot/error.log
150576 /srv/tools/shared/tools/project/wikidata-primary-sources/error.log
141724 /srv/tools/shared/tools/project/steinsplitter/error.log
137324 /srv/tools/shared/tools/project/coord/error.log
136524 /srv/tools/shared/tools/project/wd-art/error.log
118168 /srv/tools/shared/tools/project/panoviewer/error.log
110760 /srv/tools/shared/tools/project/fiwiki-tools/error.log
106316 /srv/tools/shared/tools/project/audetools/error.log
103100 /srv/tools/shared/tools/project/templatetiger/error.log
102680 /srv/tools/shared/tools/project/wikivoyage/error.log
102516 /srv/tools/shared/tools/project/para/error.log
6700 /srv/tools/shared/tools/project/timescale/error.log

Disk usage after operation:

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.5T  1.2T  86% /srv/tools
/dev/drbd3      5.0T  3.1T  1.7T  66% /srv/misc
Magnus closed subtask T208460: wikidata-todo is using 959GB in shared NFS area as Resolved.Nov 1 2018, 11:01 AM
• GTirloni added a comment.Nov 1 2018, 12:14 PM

Current disk usage:

dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  5.6T  2.1T  74% /srv/tools
/dev/drbd3      5.0T  3.1T  1.7T  66% /srv/misc
• GTirloni closed subtask T208495: User maximilianklein is using 1.2TB out of 8TB available in NFS storage as Resolved.Nov 1 2018, 5:01 PM

Current disk usage:

/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  5.6T  2.1T  74% /srv/tools
/dev/drbd3      5.0T  2.2T  2.6T  45% /srv/misc
Bstorm added a comment.Nov 19 2018, 6:47 PM

For us that's pretty good. We could probably just close this one for now.

Bstorm added a comment.Nov 19 2018, 6:47 PM

Unless we want to wait for the subtasks.

• GTirloni changed the status of subtask T208456: templatetiger is using 1.2TB out of 8T available Tools from Open to Stalled.Dec 1 2018, 2:23 AM
• GTirloni closed this task as Resolved.Dec 1 2018, 5:40 AM
• GTirloni changed the status of subtask T208466: bookworm is using 254GB of space from Open to Stalled.
• GTirloni unsubscribed.Dec 1 2018, 11:01 AM
• GTirloni closed subtask T208456: templatetiger is using 1.2TB out of 8T available Tools as Resolved.Dec 2 2018, 7:53 PM
zhuyifei1999 mentioned this in T216988: labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%):.Feb 25 2019, 9:24 AM
Bstorm mentioned this in T217993: 2019-03-10: tools and NFS share cleanup (high usage).Mar 10 2019, 10:43 PM
Bstorm changed the status of subtask T208466: bookworm is using 254GB of space from Stalled to Open.Mar 10 2019, 11:13 PM
Bstorm closed subtask T208466: bookworm is using 254GB of space as Resolved.
Bstorm mentioned this in T233120: 2019-09-17: tools share cleanup (high usage).Sep 17 2019, 4:12 PM
bd808 mentioned this in T233347: Remove access.log generation from default lighttpd.conf generated by `webservice`.Sep 19 2019, 7:04 PM
fnegri mentioned this in T323798: 2022-11-25: tools nfs share cleanup.Nov 25 2022, 8:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL