Disk usage on the tools NFS filesystem just hit 95%, triggering a page.
This ticket is to track cleanup efforts both immediately and in the tradition of T206239
Disk usage on the tools NFS filesystem just hit 95%, triggering a page.
This ticket is to track cleanup efforts both immediately and in the tradition of T206239
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | Bstorm | T217993 2019-03-10: tools and NFS share cleanup (high usage) | |||
Resolved | Cyberpower678 | T217999 Prevent iabot from filling the tools project NFS filesystems | |||
Resolved | Bstorm | T208466 bookworm is using 254GB of space | |||
Resolved | Bstorm | T122508 Prevent overly-large log files | |||
Open | None | T97861 Provide centralized logging (logstash) for Toolforge | |||
Open | None | T127367 Provide modern, non-NFS error log solution for Toolforge webservices and bots | |||
Stalled | None | T127368 Estimate hardware requirements for Toolforge logging elastic cluster | |||
Open | None | T152235 Simple logrotate service for users of Tools as stopgap before central logging |
ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_2019-03-10.txt started up on labstore1004
@Cyberpower678, it appears the iabot tool nearly took down the NFS. I have zeroed out the file at /data/project/iabot/Wokrers/Worker2.err, which won't help with troubleshooting, but there is clearly something wrong there.
That leaves tools looking pretty good.
/dev/drbd4 8.0T 5.7T 1.9T 76% /srv/tools
With the iabot ticket open to track that effort, this one can now be closed.
If things still seem high on Tuesday, we could, perhaps, reopen and create more subtasks to clean up tools.