Page MenuHomePhabricator
Create Task
Maniphest T288276

2021-08-05: Tools NFS share cleanup
Closed, ResolvedPublic
Actions

Description

The space on tools just crossed the line and tripped the alarm.

Notification Type: PROBLEM

Service: NFS Share Volume Space /srv/tools
Host: labstore1004
Address: 10.64.37.19
State: CRITICAL

Date/Time: Thu Aug 5 17:29:22 UTC 2021

Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage%23NFS_volume_cleanup https://grafana.wikimedia.org/d/50z0i4XWz/tools-overall-nfs-storage-utilization?orgId=1

Acknowledged by :

Additional Info:

DISK CRITICAL - free space: /srv/tools 1241687 MB (15% inode=78%):

In the tradition of T284964, we'll clean it up.

Actual current usage is /dev/drbd4 8.0T 6.5T 1.2T 85% /srv/tools despite what icinga thinks

Related Objects
Search...

Event Timeline

• Bstorm triaged this task as Medium priority.Aug 5 2021, 5:58 PM
• Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 5 2021, 5:58 PM
• Bstorm merged a task: T284964: 2021-06-15: Tools NFS share cleanup.Aug 5 2021, 5:58 PM
• Bstorm added a subtask: T284966: cluebotng using high storage on NFS.
• Bstorm added subscribers: RhinosF1, • nskaggs.
• Bstorm added a subtask: T284968: eatchabot using a lot of NFS storage.Aug 5 2021, 6:00 PM
• Bstorm updated the task description. (Show Details)
Peachey88 added a project: cloud-services-team (Kanban).Aug 5 2021, 8:16 PM
• Bstorm added a comment.Aug 5 2021, 9:49 PM

Found a runaway file: 995G /srv/tools/shared/tools/project/iabot/Workers/Worker2.out

Stashbot added a comment.Aug 5 2021, 9:50 PM

Mentioned in SAL (#wikimedia-cloud) [2021-08-05T21:50:56Z] <bstorm> truncated 995G Workers/Worker2.out T288276

• Bstorm added a comment.Aug 6 2021, 12:45 AM

Large NFS projects aren't too bad now that I got rid of that one file:

71G     /srv/tools/shared/tools/project/digero
74G     /srv/tools/shared/tools/project/mix-n-match
78G     /srv/tools/shared/tools/project/eatchabot
80G     /srv/tools/shared/tools/project/iabot
104G    /srv/tools/shared/tools/project/templatehoard
107G    /srv/tools/shared/tools/project/phetools
111G    /srv/tools/shared/tools/project/panoviewer
114G    /srv/tools/shared/tools/project/wikidata-analysis
174G    /srv/tools/shared/tools/project/zoomviewer
198G    /srv/tools/shared/tools/project/glamtools

That's the largest project dirs. Most of iabot is actually that file which is now stuck at over 70 GB.

Graemebp closed subtask T288307: verification-pages has a very large old log file as Resolved.Aug 6 2021, 7:09 AM
• Bstorm added a comment.Sep 8 2021, 9:34 PM

It's back up to 82%. Checking on things.

Stashbot added a comment.Sep 12 2021, 5:04 PM

Mentioned in SAL (#wikimedia-cloud) [2021-09-12T17:04:16Z] <bstorm> truncated 58gb error.log file T288276

Stashbot added a comment.Sep 12 2021, 5:06 PM

Mentioned in SAL (#wikimedia-cloud) [2021-09-12T17:06:06Z] <bstorm> truncating 45G reflinks.err file T288276

Stashbot added a comment.Sep 12 2021, 5:07 PM

Mentioned in SAL (#wikimedia-cloud) [2021-09-12T17:07:37Z] <bstorm> truncated 77G Worker2.out T288276

• Bstorm added a subtask: T278199: khanamalumat has a job that puts a lot of text in a log file when not doing any changes.Sep 12 2021, 5:28 PM
Rubin16 subscribed.Sep 13 2021, 5:53 AM
In T288276#7346987, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2021-09-12T17:06:06Z] <bstorm> truncating 45G reflinks.err file T288276

I have deleted my file and created a job to regularly clear it. Sorry for inconvenience

• Bstorm added a comment.Sep 15 2021, 4:29 PM

Ok, we are now at /dev/drbd4 8.0T 6.5T 1.2T 86% /srv/tools
:(
I'm going to run a large du across group projects, which is going to cause some iowait, but I don't see another way to get the info needed quickly.

• Bstorm added a comment.Sep 15 2021, 11:28 PM

Ok, here's what I've got for the top projects:

42G     /srv/tools/shared/tools/project/lbenedix
46G     /srv/tools/shared/tools/project/nyandata
48G     /srv/tools/shared/tools/project/splinetools
49G     /srv/tools/shared/tools/project/render
52G     /srv/tools/shared/tools/project/render-tests
54G     /srv/tools/shared/tools/project/freddy2001
56G     /srv/tools/shared/tools/project/persondata
57G     /srv/tools/shared/tools/project/ifttt
68G     /srv/tools/shared/tools/project/cluebotng
72G     /srv/tools/shared/tools/project/digero
79G     /srv/tools/shared/tools/project/mix-n-match
84G     /srv/tools/shared/tools/project/eatchabot
111G    /srv/tools/shared/tools/project/panoviewer
111G    /srv/tools/shared/tools/project/phetools
112G    /srv/tools/shared/tools/project/templatehoard
114G    /srv/tools/shared/tools/project/wikidata-analysis
180G    /srv/tools/shared/tools/project/zoomviewer
220G    /srv/tools/shared/tools/project/glamtools
524G    /srv/tools/shared/tools/project/iabot
• Bstorm added a comment.Sep 15 2021, 11:32 PM

The vast majority of that space in iabot are worker out files. It's just overly verbose logging.

Stashbot mentioned this in T288300: IAbot is writing loads of text to Toolforge NFS at a high rate.Sep 15 2021, 11:43 PM

Mentioned in SAL (#wikimedia-cloud) [2021-09-15T23:43:41Z] <bstorm> truncated massive files Worker[1-5].out T288300 T288276

• Bstorm added a comment.Sep 15 2021, 11:45 PM

That alone put us to /dev/drbd4 8.0T 6.0T 1.7T 79% /srv/tools

• Bstorm closed subtask T288300: IAbot is writing loads of text to Toolforge NFS at a high rate as Resolved.Sep 16 2021, 3:34 PM
• Bstorm closed this task as Resolved.Sep 27 2021, 8:58 PM
• Bstorm claimed this task.

Going to close this just so it isn't open forever. It's already at 80% so a new one will be needed soon.

DamianZaremba closed subtask T284966: cluebotng using high storage on NFS as Resolved.Nov 1 2021, 9:33 PM
fnegri mentioned this in T323798: 2022-11-25: tools nfs share cleanup.Nov 25 2022, 8:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL