Page MenuHomePhabricator
Create Task
Maniphest T233120

2019-09-17: tools share cleanup (high usage)
Closed, ResolvedPublic
Actions

Description

/dev/drbd4 8.0T 7.0T 607G 93% /srv/tools

We need to start another round of cleanup in toolforge.
In the tradition of linking in previous tasks for history: T206239: 2018-10-04: tools and NFS share cleanup (high usage)

Related Objects
Search...

Event Timeline

• Bstorm triaged this task as Medium priority.Sep 17 2019, 4:12 PM
• Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2019, 4:12 PM
• Bstorm raised the priority of this task from Medium to High.Sep 17 2019, 4:13 PM
• Bstorm moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
• Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.
• Bstorm assigned this task to • Phamhi.Sep 17 2019, 6:41 PM
• Phamhi added a comment.Sep 17 2019, 8:29 PM

Running ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_20190917.txt

• Phamhi added a comment.Sep 17 2019, 8:58 PM

Top 20...

29329232 KB /srv/tools/shared/tools/project/pageviews/access.log
31437236 KB /srv/tools/shared/tools/project/freebase/freebase-rdf-latest.gz
34621292 KB /srv/tools/shared/tools/project/verification-pages/verification-pages/log/production.log.1
34821024 KB /srv/tools/shared/tools/project/khanamalumat/purawiki.err
36272936 KB /srv/tools/shared/tools/project/ftl/access.log
43631092 KB /srv/tools/shared/tools/project/dewikinews-rss/error.log
47438752 KB /srv/tools/shared/tools/project/khanamalumat/qaus.err
49303704 KB /srv/tools/shared/tools/project/splinetools/dumps/enwiki-20141106-pages-articles.xml
49423488 KB /srv/tools/shared/tools/project/osm/access.log
54437988 KB /srv/tools/shared/tools/project/toolserver-home-archive/archive-2014-06-05.tar.xz
55842976 KB /srv/tools/shared/tools/project/mbrt1/cron-29.out
57829176 KB /srv/tools/shared/tools/project/.shared/dumps/20190819.json.gz
57891896 KB /srv/tools/shared/tools/project/geohack/access.log
58293516 KB /srv/tools/shared/tools/project/.shared/dumps/20190902.json.gz
59849260 KB /srv/tools/shared/tools/project/mbrt1/cron-29.err
68575372 KB /srv/tools/shared/tools/project/zoomviewer/access.log
70415608 KB /srv/tools/shared/tools/project/wikiedudashboard/access.log
78428056 KB /srv/tools/shared/tools/project/oar/repository_text_2014-06-13.tar.gz
133057456 KB /srv/tools/shared/tools/project/robokobot/virgule.err
159132952 KB /srv/tools/shared/tools/project/jarbot-ii/destfrtoar.txt
• Phamhi added a comment.Sep 17 2019, 9:23 PM

Truncated >100MB error.log files (156GB freed)

103264 KB /srv/tools/shared/tools/project/wikitools/error.log
104256 KB /srv/tools/shared/tools/project/glamtools/error.log
106724 KB /srv/tools/shared/tools/project/pltools/error.log
110884 KB /srv/tools/shared/tools/project/wsexport/error_2019-07-03.log
114720 KB /srv/tools/shared/tools/project/para/error.log
119364 KB /srv/tools/shared/tools/project/checkpersondata/error.log
129944 KB /srv/tools/shared/tools/project/dow/error.log
137932 KB /srv/tools/shared/tools/project/anagrimes/error.log
139200 KB /srv/tools/shared/tools/project/panoviewer/error.log
145680 KB /srv/tools/shared/tools/project/dupdet/error.log
153472 KB /srv/tools/shared/tools/project/dplbot/error.log
159288 KB /srv/tools/shared/tools/project/templator/error.log
175664 KB /srv/tools/shared/tools/project/ia-upload/error.log
184544 KB /srv/tools/shared/tools/project/manypedia/error.log
190752 KB /srv/tools/shared/tools/project/osm4wiki/error.log
193040 KB /srv/tools/shared/tools/project/catnap/error.log
199660 KB /srv/tools/shared/tools/project/zoomable-images/error.log
207564 KB /srv/tools/shared/tools/project/geohack/error.log.1
217060 KB /srv/tools/shared/tools/project/magnustools/error.log
239836 KB /srv/tools/shared/tools/project/admin/error.log
255696 KB /srv/tools/shared/tools/project/robin/error.log
257524 KB /srv/tools/shared/tools/project/wikidata-game/error.log
259568 KB /srv/tools/shared/tools/project/wikivoyage/error.log
282688 KB /srv/tools/shared/tools/project/enwp10/error.log.old
292136 KB /srv/tools/shared/tools/project/deadlinkfinder/error.log
300704 KB /srv/tools/shared/tools/project/reasonator/error.log
311164 KB /srv/tools/shared/tools/project/wikiedudashboard/error.log
314080 KB /srv/tools/shared/tools/project/templatetiger/error.log
322596 KB /srv/tools/shared/tools/project/slumpartikel/error.log
333928 KB /srv/tools/shared/tools/project/isprangefinder/error.log
365952 KB /srv/tools/shared/tools/project/flickr2commons/error.log
390360 KB /srv/tools/shared/tools/project/mediawiki-feeds/error.log
422168 KB /srv/tools/shared/tools/project/antigng-bot/httpd/logs/error_log
464412 KB /srv/tools/shared/tools/project/refill/error.log
516336 KB /srv/tools/shared/tools/project/featured-article/error.log
525876 KB /srv/tools/shared/tools/project/hroest/error.log
545964 KB /srv/tools/shared/tools/project/geohack/error.log
705448 KB /srv/tools/shared/tools/project/awb/error.log
867204 KB /srv/tools/shared/tools/project/expose-data/error.log
892532 KB /srv/tools/shared/tools/project/pinyin-wiki/error.log
1009192 KB /srv/tools/shared/tools/project/denkmalliste/error.log
1108152 KB /srv/tools/shared/tools/project/zoomviewer/error.log
1131412 KB /srv/tools/shared/tools/project/joanjoc/error.log
1153244 KB /srv/tools/shared/tools/project/fiwiki-tools/error.log
1309664 KB /srv/tools/shared/tools/project/jembot/error.log
1449080 KB /srv/tools/shared/tools/project/recoin/error.log
1522976 KB /srv/tools/shared/tools/project/wikidata-todo/error.log
1524360 KB /srv/tools/shared/tools/project/dnbtools/error.log
1943772 KB /srv/tools/shared/tools/project/wsexport/error_2019-05-01.log
2237420 KB /srv/tools/shared/tools/project/makeref/error.log
2402108 KB /srv/tools/shared/tools/project/query2map/error.log
2943248 KB /srv/tools/shared/tools/project/commonscategorycount/error.log
2995000 KB /srv/tools/shared/tools/project/enwp10/error.log
3019812 KB /srv/tools/shared/tools/project/ru_monuments/error.log
3150996 KB /srv/tools/shared/tools/project/admin/logs/error.log
3183264 KB /srv/tools/shared/tools/project/mix-n-match/error.log
3499280 KB /srv/tools/shared/tools/project/wp-world/error.log
4234064 KB /srv/tools/shared/tools/project/render-tests/error.log
4685956 KB /srv/tools/shared/tools/project/wikidata-primary-sources/error.log
4961164 KB /srv/tools/shared/tools/project/render/error.log
5450036 KB /srv/tools/shared/tools/project/magnus-toolserver/error.log
6535396 KB /srv/tools/shared/tools/project/request/error.log
6574168 KB /srv/tools/shared/tools/project/stimmberechtigung/error.log
9993756 KB /srv/tools/shared/tools/project/catfood/error.log
23366736 KB /srv/tools/shared/tools/project/wiwosm/error.log
43631092 KB /srv/tools/shared/tools/project/dewikinews-rss/error.log

Truncated >100MB "err$" files (456GB freed)

22624 KB /srv/tools/shared/tools/project/yifeibot/rmiw.commons.err
105088 KB /srv/tools/shared/tools/project/fatemi/ncat.err
109388 KB /srv/tools/shared/tools/project/mifterbot/cron-tools.mifterbot-commons-1.err
111088 KB /srv/tools/shared/tools/project/dexbot/en_iw.err
111320 KB /srv/tools/shared/tools/project/theoslittlebot/cite_wdl.err
111360 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20160414.import_label_from_wiki_link.ja.err
112012 KB /srv/tools/shared/tools/project/eranbot/eswiki.err
112236 KB /srv/tools/shared/tools/project/verification-pages/cron-tools.verification-pages-1.err
113936 KB /srv/tools/shared/tools/project/invadibot/inv-wes-14.err
114032 KB /srv/tools/shared/tools/project/invadibot/inv-wes-13.err
115496 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20170515.signature_check.zh-classical.err
117832 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20160915.TaiBNET.err
118564 KB /srv/tools/shared/tools/project/dykautobot/dyk.err
120088 KB /srv/tools/shared/tools/project/fist/fc_wd_depicts.err
124308 KB /srv/tools/shared/tools/project/perfectbot/Datumappend.err
126516 KB /srv/tools/shared/tools/project/rezabot/catredi.err
129516 KB /srv/tools/shared/tools/project/dimastbkbot/del_y6.err
130984 KB /srv/tools/shared/tools/project/kasparbot/kb_nt2.err
136160 KB /srv/tools/shared/tools/project/fist/fc_maintenance.err
136208 KB /srv/tools/shared/tools/project/tool-db-usage/update-cache.err
136348 KB /srv/tools/shared/tools/project/dimastbkbot/del_p6.err
136432 KB /srv/tools/shared/tools/project/archive-things-1/yt-lists.err
136904 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20160517.interlanguage_link_to_wikilinks.zh.err
137504 KB /srv/tools/shared/tools/project/archive-things-2/yt-channels-5.err
138840 KB /srv/tools/shared/tools/project/yifeibot/comsign.err
139220 KB /srv/tools/shared/tools/project/superzerocool/rate.err
141364 KB /srv/tools/shared/tools/project/cluebot3/cluebot3.err
142720 KB /srv/tools/shared/tools/project/fatemi/amarakhir.err
144204 KB /srv/tools/shared/tools/project/dimastbkbot/pat.err
144632 KB /srv/tools/shared/tools/project/avicbot/arwprd.err
144932 KB /srv/tools/shared/tools/project/framabot/frwiki-cleaner.err
145692 KB /srv/tools/shared/tools/project/isprangefinder/range-update.err
146772 KB /srv/tools/shared/tools/project/hat-collector/USRDbot.err
153396 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-20170515.signature_check.zh.err
155556 KB /srv/tools/shared/tools/project/reports-bot/logs/cron-new_discussions.err
172080 KB /srv/tools/shared/tools/project/dimastbkbot/au.err
173576 KB /srv/tools/shared/tools/project/himo/cat2018.err
174976 KB /srv/tools/shared/tools/project/zkbot/zk-stats-d.err
176580 KB /srv/tools/shared/tools/project/wdvaliditycheck/cron-checkversion.err
179500 KB /srv/tools/shared/tools/project/zkbot/zk-weekly.err
180452 KB /srv/tools/shared/tools/project/himo/newcr.err
190900 KB /srv/tools/shared/tools/project/yifeibot/rmiw.w4.err
206008 KB /srv/tools/shared/tools/project/reports-bot/logs/cron-update_members.err
207276 KB /srv/tools/shared/tools/project/yifeibot/rmiw.w1.err
211272 KB /srv/tools/shared/tools/project/himo/fixold.err
227012 KB /srv/tools/shared/tools/project/tfaprotbot/protect.err
227504 KB /srv/tools/shared/tools/project/rxy/rxy_pwb_blockpageschecker.err
239024 KB /srv/tools/shared/tools/home/datguy/afreporter.err
242928 KB /srv/tools/shared/tools/project/avicbot/trwprd.err
252828 KB /srv/tools/shared/tools/project/shuaib-bot/uploo.err
256864 KB /srv/tools/shared/tools/project/ruwn-misc/refreshfeeds.err
262920 KB /srv/tools/shared/tools/project/jackbot/WP.err
276144 KB /srv/tools/shared/tools/project/fatemi/hfnewp.err
277292 KB /srv/tools/shared/tools/project/g13bot/g13_nudge.err
306888 KB /srv/tools/shared/tools/project/wdvaliditycheck/cron-29.err
313904 KB /srv/tools/shared/tools/project/hsfbot/chk.img.err
314720 KB /srv/tools/shared/tools/project/liangent-php/php_dispatchRC_zhwiki.err
334532 KB /srv/tools/shared/tools/project/eranbot/enwiki.err
335940 KB /srv/tools/shared/tools/project/wikihistory/dewiki_update_new2.err
337968 KB /srv/tools/shared/tools/project/archive-things/yt-channels-3.err
346464 KB /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-3.err
364492 KB /srv/tools/shared/tools/project/aswnbot/wmfqua.err
365084 KB /srv/tools/shared/tools/project/fatemi/hfnobot.err
375644 KB /srv/tools/shared/tools/project/sammour/java.err
377048 KB /srv/tools/shared/tools/project/dimastbkbot/au_ws.err
379564 KB /srv/tools/shared/tools/project/zumraband/commonscat.err
385732 KB /srv/tools/shared/tools/project/persondata/dewiki_pd.err
388244 KB /srv/tools/shared/tools/project/wikihistory/dewiki_update_new1.err
394156 KB /srv/tools/shared/tools/project/rezabot/radehc.err
394832 KB /srv/tools/shared/tools/project/lahitools/main.err
397088 KB /srv/tools/shared/tools/project/shuaib-bot/urwikidata.err
403036 KB /srv/tools/shared/tools/project/chobot/log/iw.err
434020 KB /srv/tools/shared/tools/project/rluts/tgukwiki.err
444616 KB /srv/tools/shared/tools/project/perfectbot/TemplateTranslateWrapper.err
454560 KB /srv/tools/shared/tools/project/persondata/u_pd_sex.err
458688 KB /srv/tools/shared/tools/project/persondata/_process_templatedata.err
462548 KB /srv/tools/shared/tools/project/rezabot/wpgads.err
467896 KB /srv/tools/shared/tools/project/archive-things/yt-lists.err
502656 KB /srv/tools/shared/tools/project/dimastbkbot/welcome_ws.err
523448 KB /srv/tools/shared/tools/project/invadibot/inv-es-redirects.err
540788 KB /srv/tools/shared/tools/project/rchv/java.err
556996 KB /srv/tools/shared/tools/project/himo/ref33.err
563356 KB /srv/tools/shared/tools/project/matsubot/tmp/labels1.err
571288 KB /srv/tools/shared/tools/project/rezabot/red.err
626880 KB /srv/tools/shared/tools/project/urdubot/imlaa.err
642168 KB /srv/tools/shared/tools/project/vitalarticlesbot/vitalarticlesbot.err
657868 KB /srv/tools/shared/tools/project/himo/temp_cat.err
705780 KB /srv/tools/shared/tools/project/dimastbkbot/kds.err
736160 KB /srv/tools/shared/tools/project/wdpv/wdpv-run.err
739528 KB /srv/tools/shared/tools/project/referee/rc.err
744580 KB /srv/tools/shared/tools/project/archive-things/youtube-charts2.err
756544 KB /srv/tools/shared/tools/project/perfectbot/ListMovesAcrossNamespaces.err
763600 KB /srv/tools/shared/tools/project/dimastbkbot/welcome.err
811568 KB /srv/tools/shared/tools/project/checkpersondata/cron-tools.checkpersondata-1.err
831340 KB /srv/tools/shared/tools/project/archive-things-1/youtube-charts2.err
923448 KB /srv/tools/shared/tools/project/glamtools/iv_scan.err
1016916 KB /srv/tools/shared/tools/project/fatemi/hfnobot2.err
1071336 KB /srv/tools/shared/tools/project/cewbot/cron-tools.cewbot-IRC.err
1107136 KB /srv/tools/shared/tools/project/shuaib-bot/replinkk.err
1152284 KB /srv/tools/shared/tools/project/iluvatarbot/hg.err
1273720 KB /srv/tools/shared/tools/project/yemen/peopledes.err
1552824 KB /srv/tools/shared/tools/project/dimastbkbot/ar.err
1611216 KB /srv/tools/shared/tools/project/kolbert/enkolbert.err
1649880 KB /srv/tools/shared/tools/project/zumraband/recents.err
1779816 KB /srv/tools/shared/tools/project/zumraband/mukamal.err
1937848 KB /srv/tools/shared/tools/project/khanamalumat/hawai.err
2068192 KB /srv/tools/shared/tools/project/bub/worker1.err
2080916 KB /srv/tools/shared/tools/project/kolbert/commonskolbert.err
2552260 KB /srv/tools/shared/tools/project/sigma/archivebot.err
2582084 KB /srv/tools/shared/tools/project/shuaib-bot/islahh.err
2660556 KB /srv/tools/shared/tools/project/wikitanvirbot/cron-bnwp.err
2883016 KB /srv/tools/shared/tools/project/alaabot/java.err
2956208 KB /srv/tools/shared/tools/project/njsbot/cleanWKT.err
2978536 KB /srv/tools/shared/tools/project/huggle/dx.err
3028564 KB /srv/tools/shared/tools/project/persondata/infobox.err
4359240 KB /srv/tools/shared/tools/project/wikitanvirbot/as-welcome.err
4837944 KB /srv/tools/shared/tools/project/sqid/updateData.err
5269224 KB /srv/tools/shared/tools/project/kenrick95bot/idwiki-welcome-2019-02-11.err
5369516 KB /srv/tools/shared/tools/project/aivanalysis/afc-ores.err
5577020 KB /srv/tools/shared/tools/project/kenrick95bot/idwiki-welcome-2019-06-02.err
5725488 KB /srv/tools/shared/tools/project/ping08bot/mybot.err
7617360 KB /srv/tools/shared/tools/project/fshbibbot/java.err
8026652 KB /srv/tools/shared/tools/project/cydebot/cron-tools.cydebot-1.err
8826268 KB /srv/tools/shared/tools/project/rubinbot2/refs.err
9542948 KB /srv/tools/shared/tools/project/laaknortools/create_missingimages.err
10565564 KB /srv/tools/shared/tools/project/olsibot/tung.err
17219688 KB /srv/tools/shared/tools/project/aivanalysis/runstats.err
20325184 KB /srv/tools/shared/tools/project/khanamalumat/shaher.err
34821024 KB /srv/tools/shared/tools/project/khanamalumat/purawiki.err
47438752 KB /srv/tools/shared/tools/project/khanamalumat/qaus.err
59849260 KB /srv/tools/shared/tools/project/mbrt1/cron-29.err
133057456 KB /srv/tools/shared/tools/project/robokobot/virgule.err

Disk usage after operation:

Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      9.8G  535M  8.7G   6% /srv/test
/dev/drbd4      8.0T  6.4T  1.2T  85% /srv/tools
/dev/drbd3      5.0T  3.7T  1.1T  78% /srv/misc
bd808 renamed this task from 2018-09-17: tools share cleanup (high usage) to 2019-09-17: tools share cleanup (high usage).Sep 19 2019, 6:02 PM
• Phamhi closed subtask T233323: File jarbot-ii/destfrtoar.txt size is 156GB as Resolved.Sep 19 2019, 6:41 PM
bd808 mentioned this in T233347: Remove access.log generation from default lighttpd.conf generated by `webservice`.Sep 19 2019, 7:04 PM
• Phamhi closed subtask T233330: Reducing wikiedudashboard/access.log size (68GB) as Resolved.Sep 20 2019, 12:35 PM
• Phamhi closed subtask T233327: Reducing zoomviewer/access.log size (78GB) as Resolved.Sep 23 2019, 6:29 PM
• Phamhi closed subtask T233332: Reducing mbrt1/cron-29.out size (54GB) as Resolved.Sep 24 2019, 4:39 PM
Magnus closed subtask T233730: Reducing geohack/access.log size (58GB) as Resolved.Sep 25 2019, 8:29 AM
• Phamhi moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.Sep 25 2019, 3:20 PM
• Phamhi closed this task as Resolved.Sep 26 2019, 7:06 PM
• Phamhi closed subtask T233733: Reducing osm/access.log size (100GB) as Resolved.
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd4      8.0T  6.2T  1.5T  80% /srv/tools

Closing as all subtasks have completed.

bd808 mentioned this in T233979: Big usage on nfs-tools-project.svc.eqiad.wmnet.Sep 27 2019, 9:21 PM
• Bstorm mentioned this in T247315: 2020-03-10: tools and misc nfs share cleanup.Mar 10 2020, 3:36 PM
fnegri mentioned this in T323798: 2022-11-25: tools nfs share cleanup.Nov 25 2022, 8:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL