Fix alerting for disk space on the NFS servers
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Bstorm
	Aug 26 2020, 5:06 PM

Description

Right now, /srv/tools is at 91%, but icinga reports a warning for 80%. This needs to be fixed somehow.
I don't think I've ever been paged by the setup it has for the right amount.

Details

	Subject	Repo	Branch	Lines +/-
	shared-storage: add specific NFS volume monitoring for cleanups	operations/puppet	production	+32 -2
	shared-storage: add specific NFS volume monitoring for cleanups	operations/puppet	production	+33 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Bstorm	T261335 Fix alerting for disk space on the NFS servers
Resolved	Andrew	T261336 2020-08-26: tools NFS share cleanup
Resolved	• nskaggs	T248188 Zoomviewer has ~450,000 files in NFS home directory
Resolved	• Bstorm	T261338 Remove PAWS data in Tools volume
Resolved	Wurgl	T261339 persondata tool is using a large amount of NFS space -- please try to clean it up
Resolved	bennofs	T247449 wdumps custom generated dumps storage space

Event Timeline

• Bstorm triaged this task as High priority.Aug 26 2020, 5:06 PM

• Bstorm created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2020, 5:06 PM

• Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.Aug 26 2020, 5:06 PM

• Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

So the reason alerting is not useful to us for NFS volumes is that disk alerting is assumed to be part of host monitoring. That's great except it doesn't make sense for NFS volumes due to their scale (3% free means something very different on a multi-user 9 TB volume than it does on a 20 GB root vol).

I'm finishing up a monitoring class that should fix that.

Change 622655 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] shared-storage: add specific NFS volume monitoring for cleanups

https://gerrit.wikimedia.org/r/622655

gerritbot added a project: Patch-For-Review.Aug 26 2020, 7:36 PM

Change 622655 merged by Bstorm:
[operations/puppet@production] shared-storage: add specific NFS volume monitoring for cleanups

https://gerrit.wikimedia.org/r/622655

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2020, 8:10 PM

Change 622877 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] shared-storage: add specific NFS volume monitoring for cleanups

https://gerrit.wikimedia.org/r/622877

gerritbot added a project: Patch-For-Review.Aug 27 2020, 9:02 PM

• Bstorm merged a task: T218925: Investigate or create alerting/messaging around when an NFS filesystem is ready for a cleanup.Aug 27 2020, 9:33 PM

• Bstorm added subscribers: gerritbot, ArielGlenn.

Change 622877 merged by Bstorm:
[operations/puppet@production] shared-storage: add specific NFS volume monitoring for cleanups

https://gerrit.wikimedia.org/r/622877

Maintenance_bot removed a project: Patch-For-Review.Aug 28 2020, 5:10 PM

This should be fixed now

Andrew closed subtask T261336: 2020-08-26: tools NFS share cleanup as Resolved.Nov 9 2020, 6:02 PM