Page MenuHomePhabricator

cewbot k8s-20230418.fix-redirected-wikilinks-of-templates.out is unreasonably large
Closed, ResolvedPublic

Description

root@tools-nfs-2:~# du -sh /srv/tools/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out
678G	/srv/tools/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out

This is after I truncated that file on 2024-02-19 after it was about the same size. I will truncate it after filing this task. @Kanashimi, please fix your tool to generate or store much less output.

Event Timeline

I don't think the truncate worked @taavi:

tools-sgebastion-11 tools.cewbot 22:06:27 ~
ls -lh k8s-20230418.fix-redirected-wikilinks-of-templates.out
-rw-r--r-- 1 tools.cewbot tools.cewbot 831G Feb 28 22:06 k8s-20230418.fix-redirected-wikilinks-of-templates.out

I don't think the truncate worked @taavi:

tools-sgebastion-11 tools.cewbot 22:06:27 ~
ls -lh k8s-20230418.fix-redirected-wikilinks-of-templates.out
-rw-r--r-- 1 tools.cewbot tools.cewbot 831G Feb 28 22:06 k8s-20230418.fix-redirected-wikilinks-of-templates.out

Probably because the job was still running and kept the file handle open in a weird way:

tools-sgebastion-11 tools.cewbot 22:08:52 ~
toolforge jobs show k8s-20230418.fix-redirected-wikilinks-of-templates
+-------------+------------------------------------------------------------------------------------------+
| Job name:   | k8s-20230418.fix-redirected-wikilinks-of-templates                                       |
+-------------+------------------------------------------------------------------------------------------+
| Command:    | node ./wikibot/routine/20230418.Fix_redirected_wikilinks_of_templates.js use_language=zh |
+-------------+------------------------------------------------------------------------------------------+
| Job type:   | schedule: 4 2 * * 0                                                                      |
+-------------+------------------------------------------------------------------------------------------+
| Image:      | node18                                                                                   |
+-------------+------------------------------------------------------------------------------------------+
| File log:   | yes                                                                                      |
+-------------+------------------------------------------------------------------------------------------+
| Output log: | /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out              |
+-------------+------------------------------------------------------------------------------------------+
| Error log:  | /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.err              |
+-------------+------------------------------------------------------------------------------------------+
| Emails:     | none                                                                                     |
+-------------+------------------------------------------------------------------------------------------+
| Resources:  | mem: 2Gi, cpu: default                                                                   |
+-------------+------------------------------------------------------------------------------------------+
| Mounts:     | all                                                                                      |
+-------------+------------------------------------------------------------------------------------------+
| Retry:      | no                                                                                       |
+-------------+------------------------------------------------------------------------------------------+
| Status:     | Running for 17d20h4m                                                                     |
+-------------+------------------------------------------------------------------------------------------+
| Hints:      | Last run at 2024-02-20T12:46:27Z. Pod in 'Running' phase. State                          |
|             | 'running'. Started at '2024-02-20T12:47:23Z'.                                            |
+-------------+------------------------------------------------------------------------------------------+
taavi@tools-sgebastion-11:~ $ kubectl sudo delete job -n tool-cewbot k8s-20230418.fix-redirected-wikilinks-of-templates-28460284
job.batch "k8s-20230418.fix-redirected-wikilinks-of-templates-28460284" deleted
taavi@tools-nfs-2:~$ sudo truncate -s 0 /srv/tools/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out
taavi@tools-sgebastion-11:~ $ ls -lah /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out
-rw-r--r-- 1 tools.cewbot tools.cewbot 0 Feb 29 08:45 /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out
dcaro subscribed.

The file is now ~58M, I think we can close this task, the tool does not seem to generate too much output (would be nice to logrotate it though https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Pruning_log_files):

root@tools-k8s-worker-nfs-39:~# ls -lah  /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out
-rw-r--r-- 1 tools.cewbot tools.cewbot 56M Mar 10 08:36 /data/project/cewbot/k8s-20230418.fix-redirected-wikilinks-of-templates.out