This triggered again, very similar to T272247: 2021-01-17: tools NFS share cleanup.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | dcaro | T273961 2021-02-05: tools NFS share cleanup, v2 | |||
| Open | None | T273994 wikidata-todo tool copies large dump files from dumps.wikimedia.org to /shared/dumps eating up NFS space |
Event Timeline
Comment Actions
Did a quick check, following the previous task comments, it seems that there's a bot logging lots and lots of error:
root@labstore1004:~# ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%k KB %p\n" > tools_large_files_20210203.txt
root@labstore1004:~# cat tools_large_files_20210203.txt | sort -n | tail -n 10 34621292 KB /srv/tools/shared/tools/project/verification-pages/verification-pages/log/production.log.1 36285016 KB /srv/tools/shared/tools/project/wiki2prop/prediction_ranked_Wiki2PropDEPLOY_year2018_embedding300LG_DEPLOY.h5 36784272 KB /srv/tools/shared/tools/project/khanamalumat/purawiki.err 49303704 KB /srv/tools/shared/tools/project/splinetools/dumps/enwiki-20141106-pages-articles.xml 64778744 KB /srv/tools/shared/tools/project/wikidata-analysis/public_html_tmp/dumpfiles/json-20191125/20191125.json.gz 75763044 KB /srv/tools/shared/tools/project/.shared/dumps/20210201.json.gz 89481636 KB /srv/tools/shared/tools/project/.shared/dumps/20210104.json.gz 89831120 KB /srv/tools/shared/tools/project/.shared/dumps/20210118.json.gz 441113452 KB /srv/tools/shared/tools/project/meetbot/meetbot.out 441139108 KB /srv/tools/shared/tools/project/meetbot/logs/messages.log
root@labstore1004:~# tail /srv/tools/shared/tools/project/meetbot/logs/messages.log INFO 2021-02-05T16:02:33 supybot Reconnecting to freenode at 2021-02-05T16:07:33. WARNING 2021-02-05T16:02:33 supybot Scheduling a second reconnect when one is already scheduled. This is a bug; please report it, with an explanation of what caused this to happen. WARNING 2021-02-05T16:02:33 supybot Disconnect from irc.freenode.net:6667: error: [Errno 32] Broken pipe. INFO 2021-02-05T16:02:33 supybot Reconnecting to freenode at 2021-02-05T16:07:33. WARNING 2021-02-05T16:02:33 supybot Scheduling a second reconnect when one is already scheduled. This is a bug; please report it, with an explanation of what caused this to happen. WARNING 2021-02-05T16:02:33 supybot Disconnect from irc.freenode.net:6667: error: [Errno 32] Broken pipe. INFO 2021-02-05T16:02:33 supybot Reconnecting to freenode at 2021-02-05T16:07:33. WARNING 2021-02-05T16:02:33 supybot Scheduling a second reconnect when one is already scheduled. This is a bug; please report it, with an explanation of what caused this to happen. WARNING 2021-02-05T16:02:33 supybot Disconnect from irc.freenode.net:6667: error: [Errno 32] Broken pipe. INFO 2021-02-05T16:02:33 supybot Reconnecting to freenode at 2021-02-05T16:07:33.
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-02-05T17:09:55Z] <wm-bot> <bd808> Restarted bot to allow truncation of 532G log file (T273961)
Comment Actions
Sorry for the fly-by comment, I just happened to see this:
64778744 KB /srv/tools/shared/tools/project/wikidata-analysis/public_html_tmp/dumpfiles/json-20191125/20191125.json.gz
@Addshore says that “everything in /srv/tools/shared/tools/project/wikidata-analysis/public_html_tmp/dumpfiles can be nuked periodically”, so feel free to delete that file to clean up some more space. (I’m not a member of the project, so I can’t do it myself.)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-02-05T18:06:36Z] <wm-bot> <bd808> Restarted bot to allow truncation of another 532G log file (T273961)