Disk usage on the tools NFS filesystem just hit 95%, triggering a page.
This ticket is to track cleanup efforts both immediately and in the tradition of T206239
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Bstorm | T217993 2019-03-10: tools and NFS share cleanup (high usage) | |||
| Resolved | Cyberpower678 | T217999 Prevent iabot from filling the tools project NFS filesystems | |||
| Resolved | Bstorm | T208466 bookworm is using 254GB of space | |||
| Resolved | Bstorm | T122508 Prevent overly-large log files | |||
| Open | None | T97861 Provide centralized logging (logstash) for Toolforge | |||
| Open | None | T127367 Provide modern, non-NFS error log solution for Toolforge webservices and bots | |||
| Stalled | None | T127368 Estimate hardware requirements for Toolforge logging elastic cluster | |||
| Open | None | T152235 Simple logrotate service for users of Tools as stopgap before central logging |
Event Timeline
Comment Actions
ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_2019-03-10.txt started up on labstore1004
Comment Actions
@Cyberpower678, it appears the iabot tool nearly took down the NFS. I have zeroed out the file at /data/project/iabot/Wokrers/Worker2.err, which won't help with troubleshooting, but there is clearly something wrong there.
Comment Actions
That leaves tools looking pretty good.
/dev/drbd4 8.0T 5.7T 1.9T 76% /srv/tools
With the iabot ticket open to track that effort, this one can now be closed.
Comment Actions
If things still seem high on Tuesday, we could, perhaps, reopen and create more subtasks to clean up tools.