2019-03-10: tools and NFS share cleanup (high usage)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Bstorm
	Mar 10 2019, 10:43 PM

Description

Disk usage on the tools NFS filesystem just hit 95%, triggering a page.

This ticket is to track cleanup efforts both immediately and in the tradition of T206239

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		• Bstorm	T217993 2019-03-10: tools and NFS share cleanup (high usage)
Resolved		Cyberpower678	T217999 Prevent iabot from filling the tools project NFS filesystems
Resolved		• Bstorm	T208466 bookworm is using 254GB of space
Resolved		• Bstorm	T122508 Prevent overly-large log files
Open		None	T97861 [toolforge.infra] Provide centralized logging (logstash) for Toolforge
Open		None	T127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge webservices and bots
Declined		None	T127368 Estimate hardware requirements for Toolforge logging elastic cluster
Open		None	T152235 Simple logrotate service for users of Tools as stopgap before central logging
Resolved	Feature	Raymond_Ndibe	T302211 toolforge-jobs: merge stdout/stderr output
Duplicate	Feature	Raymond_Ndibe	T304421 Allow customizing the out/err files with toolforge-jobs
Resolved	Feature	Raymond_Ndibe	T301901 Allow specifying the path for log files for jobs executed on the new toolforge Jobs framework
Resolved		None	T327165 toolforge-jobs: add logrotate
Resolved	Feature	taavi	T330715 Allow TJF job logs to go to Kuberenetes output buffer rather than disk
Resolved		taavi	T336057 Add commands to `webservice` and `jobs` to query logs from Kubernetes

Event Timeline

• Bstorm created this task.Mar 10 2019, 10:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2019, 10:43 PM

ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%p %k KB\n" > /root/tools_large_files_2019-03-10.txt started up on labstore1004

Sorted output in /root/tools_large_files_2019-03-10-sorted.txt

Truncating this file: 1.2T /srv/tools/shared/tools/project/iabot/Workers/Worker2.err

That brought it to /dev/drbd4 8.0T 6.0T 1.7T 79% /srv/tools

@Cyberpower678, it appears the iabot tool nearly took down the NFS. I have zeroed out the file at /data/project/iabot/Wokrers/Worker2.err, which won't help with troubleshooting, but there is clearly something wrong there.

I'll create a subtask to address that particular issue.

• Bstorm added a subtask: T208466: bookworm is using 254GB of space.Mar 10 2019, 11:11 PM

• Bstorm changed the status of subtask T208466: bookworm is using 254GB of space from Stalled to Open.

That leaves tools looking pretty good.

/dev/drbd4      8.0T  5.7T  1.9T  76% /srv/tools

With the iabot ticket open to track that effort, this one can now be closed.

If things still seem high on Tuesday, we could, perhaps, reopen and create more subtasks to clean up tools.

Cyberpower678 closed subtask T217999: Prevent iabot from filling the tools project NFS filesystems as Resolved.Mar 12 2019, 6:33 PM

• Bstorm added a subtask: T122508: Prevent overly-large log files.Mar 12 2019, 7:00 PM

• Bstorm closed subtask T122508: Prevent overly-large log files as Resolved.Mar 14 2019, 6:06 PM

fnegri mentioned this in T323798: 2022-11-25: tools nfs share cleanup.Nov 25 2022, 8:43 AM

2019-03-10: tools and NFS share cleanup (high usage)Closed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

2019-03-10: tools and NFS share cleanup (high usage)
Closed, ResolvedPublic
Actions

Related Objects
Search...