Page MenuHomePhabricator

2019-03-10: tools and NFS share cleanup (high usage)
Closed, ResolvedPublic

Description

Disk usage on the tools NFS filesystem just hit 95%, triggering a page.

This ticket is to track cleanup efforts both immediately and in the tradition of T206239

Event Timeline

Bstorm triaged this task as High priority.Mar 10 2019, 10:43 PM

ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_2019-03-10.txt started up on labstore1004

Sorted output in /root/tools_large_files_2019-03-10-sorted.txt

Truncating this file: 1.2T /srv/tools/shared/tools/project/iabot/Workers/Worker2.err

That brought it to /dev/drbd4 8.0T 6.0T 1.7T 79% /srv/tools

@Cyberpower678, it appears the iabot tool nearly took down the NFS. I have zeroed out the file at /data/project/iabot/Wokrers/Worker2.err, which won't help with troubleshooting, but there is clearly something wrong there.

I'll create a subtask to address that particular issue.

That leaves tools looking pretty good.

/dev/drbd4      8.0T  5.7T  1.9T  76% /srv/tools

With the iabot ticket open to track that effort, this one can now be closed.

Bstorm claimed this task.

If things still seem high on Tuesday, we could, perhaps, reopen and create more subtasks to clean up tools.