Page MenuHomePhabricator

2019-09-17: tools share cleanup (high usage)
Closed, ResolvedPublic

Description

/dev/drbd4 8.0T 7.0T 607G 93% /srv/tools

We need to start another round of cleanup in toolforge.
In the tradition of linking in previous tasks for history: T206239: 2018-10-04: tools and NFS share cleanup (high usage)

Event Timeline

Bstorm triaged this task as Normal priority.Sep 17 2019, 4:12 PM
Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2019, 4:12 PM
Bstorm raised the priority of this task from Normal to High.Sep 17 2019, 4:13 PM
Bstorm moved this task from Inbox to Important on the cloud-services-team (Kanban) board.
Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.
Bstorm assigned this task to Phamhi.Sep 17 2019, 6:41 PM

Running ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_20190917.txt

Top 20...

29329232 KB /srv/tools/shared/tools/project/pageviews/access.log
31437236 KB /srv/tools/shared/tools/project/freebase/freebase-rdf-latest.gz
34621292 KB /srv/tools/shared/tools/project/verification-pages/verification-pages/log/production.log.1
34821024 KB /srv/tools/shared/tools/project/khanamalumat/purawiki.err
36272936 KB /srv/tools/shared/tools/project/ftl/access.log
43631092 KB /srv/tools/shared/tools/project/dewikinews-rss/error.log
47438752 KB /srv/tools/shared/tools/project/khanamalumat/qaus.err
49303704 KB /srv/tools/shared/tools/project/splinetools/dumps/enwiki-20141106-pages-articles.xml
49423488 KB /srv/tools/shared/tools/project/osm/access.log
54437988 KB /srv/tools/shared/tools/project/toolserver-home-archive/archive-2014-06-05.tar.xz
55842976 KB /srv/tools/shared/tools/project/mbrt1/cron-29.out
57829176 KB /srv/tools/shared/tools/project/.shared/dumps/20190819.json.gz
57891896 KB /srv/tools/shared/tools/project/geohack/access.log
58293516 KB /srv/tools/shared/tools/project/.shared/dumps/20190902.json.gz
59849260 KB /srv/tools/shared/tools/project/mbrt1/cron-29.err
68575372 KB /srv/tools/shared/tools/project/zoomviewer/access.log
70415608 KB /srv/tools/shared/tools/project/wikiedudashboard/access.log
78428056 KB /srv/tools/shared/tools/project/oar/repository_text_2014-06-13.tar.gz
133057456 KB /srv/tools/shared/tools/project/robokobot/virgule.err
159132952 KB /srv/tools/shared/tools/project/jarbot-ii/destfrtoar.txt

Truncated >100MB error.log files (156GB freed)

103264 KB /srv/tools/shared/tools/project/wikitools/error.log
104256 KB /srv/tools/shared/tools/project/glamtools/error.log
106724 KB /srv/tools/shared/tools/project/pltools/error.log
110884 KB /srv/tools/shared/tools/project/wsexport/error_2019-07-03.log
114720 KB /srv/tools/shared/tools/project/para/error.log
119364 KB /srv/tools/shared/tools/project/checkpersondata/error.log
129944 KB /srv/tools/shared/tools/project/dow/error.log
137932 KB /srv/tools/shared/tools/project/anagrimes/error.log
139200 KB /srv/tools/shared/tools/project/panoviewer/error.log
145680 KB /srv/tools/shared/tools/project/dupdet/error.log
153472 KB /srv/tools/shared/tools/project/dplbot/error.log
159288 KB /srv/tools/shared/tools/project/templator/error.log
175664 KB /srv/tools/shared/tools/project/ia-upload/error.log
184544 KB /srv/tools/shared/tools/project/manypedia/error.log
190752 KB /srv/tools/shared/tools/project/osm4wiki/error.log
193040 KB /srv/tools/shared/tools/project/catnap/error.log
199660 KB /srv/tools/shared/tools/project/zoomable-images/error.log
207564 KB /srv/tools/shared/tools/project/geohack/error.log.1
217060 KB /srv/tools/shared/tools/project/magnustools/error.log
239836 KB /srv/tools/shared/tools/project/admin/error.log
255696 KB /srv/tools/shared/tools/project/robin/error.log
257524 KB /srv/tools/shared/tools/project/wikidata-game/error.log
259568 KB /srv/tools/shared/tools/project/wikivoyage/error.log
282688 KB /srv/tools/shared/tools/project/enwp10/error.log.old
292136 KB /srv/tools/shared/tools/project/deadlinkfinder/error.log
300704 KB /srv/tools/shared/tools/project/reasonator/error.log
311164 KB /srv/tools/shared/tools/project/wikiedudashboard/error.log
314080 KB /srv/tools/shared/tools/project/templatetiger/error.log
322596 KB /srv/tools/shared/tools/project/slumpartikel/error.log
333928 KB /srv/tools/shared/tools/project/isprangefinder/error.log
365952 KB /srv/tools/shared/tools/project/flickr2commons/error.log
390360 KB /srv/tools/shared/tools/project/mediawiki-feeds/error.log
422168 KB /srv/tools/shared/tools/project/antigng-bot/httpd/logs/error_log
464412 KB /srv/tools/shared/tools/project/refill/error.log
516336 KB /srv/tools/shared/tools/project/featured-article/error.log
525876 KB /srv/tools/shared/tools/project/hroest/error.log
545964 KB /srv/tools/shared/tools/project/geohack/error.log
705448 KB /srv/tools/shared/tools/project/awb/error.log
867204 KB /srv/tools/shared/tools/project/expose-data/error.log
892532 KB /srv/tools/shared/tools/project/pinyin-wiki/error.log
1009192 KB /srv/tools/shared/tools/project/denkmalliste/error.log
1108152 KB /srv/tools/shared/tools/project/zoomviewer/error.log
1131412 KB /srv/tools/shared/tools/project/joanjoc/error.log
1153244 KB /srv/tools/shared/tools/project/fiwiki-tools/error.log
1309664 KB /srv/tools/shared/tools/project/jembot/error.log
1449080 KB /srv/tools/shared/tools/project/recoin/error.log
1522976 KB /srv/tools/shared/tools/project/wikidata-todo/error.log
1524360 KB /srv/tools/shared/tools/project/dnbtools/error.log
1943772 KB /srv/tools/shared/tools/project/wsexport/error_2019-05-01.log
2237420 KB /srv/tools/shared/tools/project/makeref/error.log
2402108 KB /srv/tools/shared/tools/project/query2map/error.log
2943248 KB /srv/tools/shared/tools/project/commonscategorycount/error.log
2995000 KB /srv/tools/shared/tools/project/enwp10/error.log
3019812 KB /srv/tools/shared/tools/project/ru_monuments/error.log
3150996 KB /srv/tools/shared/tools/project/admin/logs/error.log
3183264 KB /srv/tools/shared/tools/project/mix-n-match/error.log
3499280 KB /srv/tools/shared/tools/project/wp-world/error.log
4234064 KB /srv/tools/shared/tools/project/render-tests/error.log
4685956 KB /srv/tools/shared/tools/project/wikidata-primary-sources/error.log
4961164 KB /srv/tools/shared/tools/project/render/error.log
5450036 KB /srv/tools/shared/tools/project/magnus-toolserver/error.log
6535396 KB /srv/tools/shared/tools/project/request/error.log
6574168 KB /srv/tools/shared/tools/project/stimmberechtigung/error.log
9993756 KB /srv/tools/shared/tools/project/catfood/error.log
23366736 KB /srv/tools/shared/tools/project/wiwosm/error.log
43631092 KB /srv/tools/shared/tools/project/dewikinews-rss/error.log

Truncated >100MB "err$" files (456GB freed)

22624 KB /srv/tools/shared/tools/project/yifeibot/rmiw.commons.err
105088 KB /srv/tools/shared/tools/project/fatemi/ncat.err
109388 KB /srv/tools/shared/tools/project/mifterbot/cron-tools.mifterbot-commons-1.err
111088 KB /srv/tools/shared/tools/project/dexbot/en_iw.err
111320 KB /srv/tools/shared/tools/project/theoslittlebot/cite_wdl.err
111360 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20160414.import_label_from_wiki_link.ja.err
112012 KB /srv/tools/shared/tools/project/eranbot/eswiki.err
112236 KB /srv/tools/shared/tools/project/verification-pages/cron-tools.verification-pages-1.err
113936 KB /srv/tools/shared/tools/project/invadibot/inv-wes-14.err
114032 KB /srv/tools/shared/tools/project/invadibot/inv-wes-13.err
115496 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20170515.signature_check.zh-classical.err
117832 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20160915.TaiBNET.err
118564 KB /srv/tools/shared/tools/project/dykautobot/dyk.err
120088 KB /srv/tools/shared/tools/project/fist/fc_wd_depicts.err
124308 KB /srv/tools/shared/tools/project/perfectbot/Datumappend.err
126516 KB /srv/tools/shared/tools/project/rezabot/catredi.err
129516 KB /srv/tools/shared/tools/project/dimastbkbot/del_y6.err
130984 KB /srv/tools/shared/tools/project/kasparbot/kb_nt2.err
136160 KB /srv/tools/shared/tools/project/fist/fc_maintenance.err
136208 KB /srv/tools/shared/tools/project/tool-db-usage/update-cache.err
136348 KB /srv/tools/shared/tools/project/dimastbkbot/del_p6.err
136432 KB /srv/tools/shared/tools/project/archive-things-1/yt-lists.err
136904 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20160517.interlanguage_link_to_wikilinks.zh.err
137504 KB /srv/tools/shared/tools/project/archive-things-2/yt-channels-5.err
138840 KB /srv/tools/shared/tools/project/yifeibot/comsign.err
139220 KB /srv/tools/shared/tools/project/superzerocool/rate.err
141364 KB /srv/tools/shared/tools/project/cluebot3/cluebot3.err
142720 KB /srv/tools/shared/tools/project/fatemi/amarakhir.err
144204 KB /srv/tools/shared/tools/project/dimastbkbot/pat.err
144632 KB /srv/tools/shared/tools/project/avicbot/arwprd.err
144932 KB /srv/tools/shared/tools/project/framabot/frwiki-cleaner.err
145692 KB /srv/tools/shared/tools/project/isprangefinder/range-update.err
146772 KB /srv/tools/shared/tools/project/hat-collector/USRDbot.err
153396 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20170515.signature_check.zh.err
155556 KB /srv/tools/shared/tools/project/reports-bot/logs/cron-new_discussions.err
172080 KB /srv/tools/shared/tools/project/dimastbkbot/au.err
173576 KB /srv/tools/shared/tools/project/himo/cat2018.err
174976 KB /srv/tools/shared/tools/project/zkbot/zk-stats-d.err
176580 KB /srv/tools/shared/tools/project/wdvaliditycheck/cron-checkversion.err
179500 KB /srv/tools/shared/tools/project/zkbot/zk-weekly.err
180452 KB /srv/tools/shared/tools/project/himo/newcr.err
190900 KB /srv/tools/shared/tools/project/yifeibot/rmiw.w4.err
206008 KB /srv/tools/shared/tools/project/reports-bot/logs/cron-update_members.err
207276 KB /srv/tools/shared/tools/project/yifeibot/rmiw.w1.err
211272 KB /srv/tools/shared/tools/project/himo/fixold.err
227012 KB /srv/tools/shared/tools/project/tfaprotbot/protect.err
227504 KB /srv/tools/shared/tools/project/rxy/rxy_pwb_blockpageschecker.err
239024 KB /srv/tools/shared/tools/home/datguy/afreporter.err
242928 KB /srv/tools/shared/tools/project/avicbot/trwprd.err
252828 KB /srv/tools/shared/tools/project/shuaib-bot/uploo.err
256864 KB /srv/tools/shared/tools/project/ruwn-misc/refreshfeeds.err
262920 KB /srv/tools/shared/tools/project/jackbot/WP.err
276144 KB /srv/tools/shared/tools/project/fatemi/hfnewp.err
277292 KB /srv/tools/shared/tools/project/g13bot/g13_nudge.err
306888 KB /srv/tools/shared/tools/project/wdvaliditycheck/cron-29.err
313904 KB /srv/tools/shared/tools/project/hsfbot/chk.img.err
314720 KB /srv/tools/shared/tools/project/liangent-php/php_dispatchRC_zhwiki.err
334532 KB /srv/tools/shared/tools/project/eranbot/enwiki.err
335940 KB /srv/tools/shared/tools/project/wikihistory/dewiki_update_new2.err
337968 KB /srv/tools/shared/tools/project/archive-things/yt-channels-3.err
346464 KB /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-3.err
364492 KB /srv/tools/shared/tools/project/aswnbot/wmfqua.err
365084 KB /srv/tools/shared/tools/project/fatemi/hfnobot.err
375644 KB /srv/tools/shared/tools/project/sammour/java.err
377048 KB /srv/tools/shared/tools/project/dimastbkbot/au_ws.err
379564 KB /srv/tools/shared/tools/project/zumraband/commonscat.err
385732 KB /srv/tools/shared/tools/project/persondata/dewiki_pd.err
388244 KB /srv/tools/shared/tools/project/wikihistory/dewiki_update_new1.err
394156 KB /srv/tools/shared/tools/project/rezabot/radehc.err
394832 KB /srv/tools/shared/tools/project/lahitools/main.err
397088 KB /srv/tools/shared/tools/project/shuaib-bot/urwikidata.err
403036 KB /srv/tools/shared/tools/project/chobot/log/iw.err
434020 KB /srv/tools/shared/tools/project/rluts/tgukwiki.err
444616 KB /srv/tools/shared/tools/project/perfectbot/TemplateTranslateWrapper.err
454560 KB /srv/tools/shared/tools/project/persondata/u_pd_sex.err
458688 KB /srv/tools/shared/tools/project/persondata/_process_templatedata.err
462548 KB /srv/tools/shared/tools/project/rezabot/wpgads.err
467896 KB /srv/tools/shared/tools/project/archive-things/yt-lists.err
502656 KB /srv/tools/shared/tools/project/dimastbkbot/welcome_ws.err
523448 KB /srv/tools/shared/tools/project/invadibot/inv-es-redirects.err
540788 KB /srv/tools/shared/tools/project/rchv/java.err
556996 KB /srv/tools/shared/tools/project/himo/ref33.err
563356 KB /srv/tools/shared/tools/project/matsubot/tmp/labels1.err
571288 KB /srv/tools/shared/tools/project/rezabot/red.err
626880 KB /srv/tools/shared/tools/project/urdubot/imlaa.err
642168 KB /srv/tools/shared/tools/project/vitalarticlesbot/vitalarticlesbot.err
657868 KB /srv/tools/shared/tools/project/himo/temp_cat.err
705780 KB /srv/tools/shared/tools/project/dimastbkbot/kds.err
736160 KB /srv/tools/shared/tools/project/wdpv/wdpv-run.err
739528 KB /srv/tools/shared/tools/project/referee/rc.err
744580 KB /srv/tools/shared/tools/project/archive-things/youtube-charts2.err
756544 KB /srv/tools/shared/tools/project/perfectbot/ListMovesAcrossNamespaces.err
763600 KB /srv/tools/shared/tools/project/dimastbkbot/welcome.err
811568 KB /srv/tools/shared/tools/project/checkpersondata/cron-tools.checkpersondata-1.err
831340 KB /srv/tools/shared/tools/project/archive-things-1/youtube-charts2.err
923448 KB /srv/tools/shared/tools/project/glamtools/iv_scan.err
1016916 KB /srv/tools/shared/tools/project/fatemi/hfnobot2.err
1071336 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-IRC.err
1107136 KB /srv/tools/shared/tools/project/shuaib-bot/replinkk.err
1152284 KB /srv/tools/shared/tools/project/iluvatarbot/hg.err
1273720 KB /srv/tools/shared/tools/project/yemen/peopledes.err
1552824 KB /srv/tools/shared/tools/project/dimastbkbot/ar.err
1611216 KB /srv/tools/shared/tools/project/kolbert/enkolbert.err
1649880 KB /srv/tools/shared/tools/project/zumraband/recents.err
1779816 KB /srv/tools/shared/tools/project/zumraband/mukamal.err
1937848 KB /srv/tools/shared/tools/project/khanamalumat/hawai.err
2068192 KB /srv/tools/shared/tools/project/bub/worker1.err
2080916 KB /srv/tools/shared/tools/project/kolbert/commonskolbert.err
2552260 KB /srv/tools/shared/tools/project/sigma/archivebot.err
2582084 KB /srv/tools/shared/tools/project/shuaib-bot/islahh.err
2660556 KB /srv/tools/shared/tools/project/wikitanvirbot/cron-bnwp.err
2883016 KB /srv/tools/shared/tools/project/alaabot/java.err
2956208 KB /srv/tools/shared/tools/project/njsbot/cleanWKT.err
2978536 KB /srv/tools/shared/tools/project/huggle/dx.err
3028564 KB /srv/tools/shared/tools/project/persondata/infobox.err
4359240 KB /srv/tools/shared/tools/project/wikitanvirbot/as-welcome.err
4837944 KB /srv/tools/shared/tools/project/sqid/updateData.err
5269224 KB /srv/tools/shared/tools/project/kenrick95bot/idwiki-welcome-2019-02-11.err
5369516 KB /srv/tools/shared/tools/project/aivanalysis/afc-ores.err
5577020 KB /srv/tools/shared/tools/project/kenrick95bot/idwiki-welcome-2019-06-02.err
5725488 KB /srv/tools/shared/tools/project/ping08bot/mybot.err
7617360 KB /srv/tools/shared/tools/project/fshbibbot/java.err
8026652 KB /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-1.err
8826268 KB /srv/tools/shared/tools/project/rubinbot2/refs.err
9542948 KB /srv/tools/shared/tools/project/laaknortools/create_missingimages.err
10565564 KB /srv/tools/shared/tools/project/olsibot/tung.err
17219688 KB /srv/tools/shared/tools/project/aivanalysis/runstats.err
20325184 KB /srv/tools/shared/tools/project/khanamalumat/shaher.err
34821024 KB /srv/tools/shared/tools/project/khanamalumat/purawiki.err
47438752 KB /srv/tools/shared/tools/project/khanamalumat/qaus.err
59849260 KB /srv/tools/shared/tools/project/mbrt1/cron-29.err
133057456 KB /srv/tools/shared/tools/project/robokobot/virgule.err

Disk usage after operation:

Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.4T  1.2T  85% /srv/tools
/dev/drbd3      5.0T  3.7T  1.1T  78% /srv/misc
bd808 renamed this task from 2018-09-17: tools share cleanup (high usage) to 2019-09-17: tools share cleanup (high usage).Sep 19 2019, 6:02 PM
Phamhi closed this task as Resolved.Thu, Sep 26, 7:06 PM
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd4      8.0T  6.2T  1.5T  80% /srv/tools

Closing as all subtasks have completed.