Page MenuHomePhabricator

2019-03-10: tools and NFS share cleanup (high usage)
Closed, ResolvedPublic

Description

Disk usage on the tools NFS filesystem just hit 95%, triggering a page.

This ticket is to track cleanup efforts both immediately and in the tradition of T206239

Event Timeline

Bstorm created this task.Mar 10 2019, 10:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2019, 10:43 PM
Bstorm triaged this task as High priority.Mar 10 2019, 10:43 PM

ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_2019-03-10.txt started up on labstore1004

Sorted output in /root/tools_large_files_2019-03-10-sorted.txt

Truncating this file: 1.2T /srv/tools/shared/tools/project/iabot/Workers/Worker2.err

That brought it to /dev/drbd4 8.0T 6.0T 1.7T 79% /srv/tools

@Cyberpower678, it appears the iabot tool nearly took down the NFS. I have zeroed out the file at /data/project/iabot/Wokrers/Worker2.err, which won't help with troubleshooting, but there is clearly something wrong there.

I'll create a subtask to address that particular issue.

That leaves tools looking pretty good.

/dev/drbd4      8.0T  5.7T  1.9T  76% /srv/tools

With the iabot ticket open to track that effort, this one can now be closed.

Bstorm closed this task as Resolved.Mar 10 2019, 11:16 PM
Bstorm claimed this task.

If things still seem high on Tuesday, we could, perhaps, reopen and create more subtasks to clean up tools.