root@tools-nfs-2:~# du -sh /srv/tools/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out 678G /srv/tools/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out
This is after I truncated that file on 2024-02-19 after it was about the same size. I will truncate it after filing this task. @Kanashimi, please fix your tool to generate or store much less output.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T358554 2024-02-27: toolforge NFS cleanup | |||
| Resolved | Kanashimi | T358555 cewbot k8s-20230418.fix-redirected-wikilinks-of-templates.out is unreasonably large |
Mentioned in SAL (#wikimedia-cloud) [2024-02-27T08:51:12Z] <taavi> truncate 678G log file T358555
I don't think the truncate worked @taavi:
tools-sgebastion-11 tools.cewbot 22:06:27 ~ ls -lh k8s-20230418.fix-redirected-wikilinks-of-templates.out -rw-r--r-- 1 tools.cewbot tools.cewbot 831G Feb 28 22:06 k8s-20230418.fix-redirected-wikilinks-of-templates.out
Probably because the job was still running and kept the file handle open in a weird way:
tools-sgebastion-11 tools.cewbot 22:08:52 ~ toolforge jobs show k8s-20230418.fix-redirected-wikilinks-of-templates +-------------+------------------------------------------------------------------------------------------+ | Job name: | k8s-20230418.fix-redirected-wikilinks-of-templates | +-------------+------------------------------------------------------------------------------------------+ | Command: | node ./wikibot/routine/20230418.Fix_redirected_wikilinks_of_templates.js use_language=zh | +-------------+------------------------------------------------------------------------------------------+ | Job type: | schedule: 4 2 * * 0 | +-------------+------------------------------------------------------------------------------------------+ | Image: | node18 | +-------------+------------------------------------------------------------------------------------------+ | File log: | yes | +-------------+------------------------------------------------------------------------------------------+ | Output log: | /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out | +-------------+------------------------------------------------------------------------------------------+ | Error log: | /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.err | +-------------+------------------------------------------------------------------------------------------+ | Emails: | none | +-------------+------------------------------------------------------------------------------------------+ | Resources: | mem: 2Gi, cpu: default | +-------------+------------------------------------------------------------------------------------------+ | Mounts: | all | +-------------+------------------------------------------------------------------------------------------+ | Retry: | no | +-------------+------------------------------------------------------------------------------------------+ | Status: | Running for 17d20h4m | +-------------+------------------------------------------------------------------------------------------+ | Hints: | Last run at 2024-02-20T12:46:27Z. Pod in 'Running' phase. State | | | 'running'. Started at '2024-02-20T12:47:23Z'. | +-------------+------------------------------------------------------------------------------------------+
taavi@tools-sgebastion-11:~ $ kubectl sudo delete job -n tool-cewbot k8s-20230418.fix-redirected-wikilinks-of-templates-28460284 job.batch "k8s-20230418.fix-redirected-wikilinks-of-templates-28460284" deleted taavi@tools-nfs-2:~$ sudo truncate -s 0 /srv/tools/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out taavi@tools-sgebastion-11:~ $ ls -lah /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out -rw-r--r-- 1 tools.cewbot tools.cewbot 0 Feb 29 08:45 /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out
The file is now ~58M, I think we can close this task, the tool does not seem to generate too much output (would be nice to logrotate it though https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Pruning_log_files):
root@tools-k8s-worker-nfs-39:~# ls -lah /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out -rw-r--r-- 1 tools.cewbot tools.cewbot 56M Mar 10 08:36 /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out