Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | taavi | T349895 tools-nfs-2 almost out of disk space (October 2023 edition) | |||
| Resolved | None | T358554 2024-02-27: toolforge NFS cleanup | |||
| Resolved | Hoi | T349913 'hoiscript' tool uses an unreasonable amount of disk space |
Event Timeline
I have truncated the /data/project/hoiscript/ndlcrawl.err log file which was 225G alone. Please reduce the logging verbosity or add a cleanup job to ensure it doesn't grow that huge again. In addition to that, most of the disk space usage is taken by 20K PDF files /data/project/hoiscript/public_html/wzlib, please clean those up.
Mentioned in SAL (#wikimedia-cloud) [2023-10-27T14:34:24Z] <taavi> truncate 225G log file /srv/tools/project/hoiscript/ndlcrawl.err T349913
I have deleted the stale logs. The 20k PDF are public domain books to be uploaded to Commons. I think it would take a few more days to complete.
4 months later there still appear to be 533G pf PDFs in $HOME/public_html/wzlib.
bd808@tools-nfs-2:/srv/tools/project/hoiscript$ sudo du -sh . 585G . bd808@tools-nfs-2:/srv/tools/project/hoiscript$ sudo du -sh * | sort -h | tail -10 87M upload2.before地方志and家譜.err 96M upload1.before民國文獻.err 102M upload2.before民國文獻.err 106M upload4.err 116M upload1.before地方志and家譜.err 120M upload5.before民國文獻.err 171M ynutcmpd 348M ndlpd 2.8G nlcpd 580G public_html bd808@tools-nfs-2:/srv/tools/project/hoiscript$ cd public_html/ bd808@tools-nfs-2:/srv/tools/project/hoiscript/public_html$ sudo du -sh * | sort -h | tail -10 4.0K ndl.jsonl 188K ynutcm 1.2G 6533_011.pdf 9.0G nlc 17G ndlcrawl.err.zst 21G ndl 533G wzlib
@Hoi I can still see the 533G PDFs, can you please clean those up? It has been several months already since they were supposed to have been processed. Thanks!
root@tools-nfs-2:/srv/tools/project/hoiscript# du -hs public_html/* | sort -h 4.0K public_html/ndl.jsonl 188K public_html/ynutcm 9.0G public_html/nlc 17G public_html/ndlcrawl.err.zst 21G public_html/ndl 533G public_html/wzlib
@Hoi The files are still there, I'm guessing you did not have the time? Are you encountering any errors?
droot@tools-nfs-2:~# du -hs /srv/tools/project/hoiscript/public_html/* | sort -h 4.0K /srv/tools/project/hoiscript/public_html/ndl.jsonl 188K /srv/tools/project/hoiscript/public_html/ynutcm 9.0G /srv/tools/project/hoiscript/public_html/nlc 17G /srv/tools/project/hoiscript/public_html/ndlcrawl.err.zst 21G /srv/tools/project/hoiscript/public_html/ndl 533G /srv/tools/project/hoiscript/public_html/wzlib
Metadata of these files are not well-structured. I was managing to tidy them up so that template fields can be filled properly and appropriate categories can be created, which makes me frustrated. This is the reason why I leave them there so long.
I finally decide to just upload them as is. The script is now uploading them to Commons and deleting then one by one.
I found and killed a sha1sum process running on tools-sgebastion-10.tools.eqiad1.wikimedia.cloud a few minutes ago that seemed to be attempting to compute the sha1 of all 533G of PDFs here in one go. That is not something that can reasonably happen on a shared bastion server. Please rethink your process.
@bd808 I find that there are some repeated files in the folder. That was my attempt to deduplicate them in order to make the upload process more efficient. I will just let the upload script work smoothly. Sorry for any troubles caused.