Page MenuHomePhabricator

cronspam from prometheus-directory-size (on labstore1004)
Closed, ResolvedPublic

Description

There is a prometheus-directory-size exporter at:

profile::prometheus::node_directorysize
modules/profile/templates/prometheus/node-directory-size.erb

On labstore1004 it creates daily cronspam due to:

find: ‘/srv/tools/shared/tools/project/paws/userhomes/*’: No such file or directory

Not sure if the proper fix is to ignore that path above or to generally make sure it does not alert when a directory does not exist but probably BOTH.

Would be nice to stop the daily mails one way or the other.

also see: T211861

Event Timeline

Dzahn created this task.Nov 20 2020, 4:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2020, 4:59 PM
Dzahn renamed this task from cronspam from prometheus-directory-size to cronspam from prometheus-directory-size (on labstore1004).Nov 20 2020, 5:02 PM
Dzahn updated the task description. (Show Details)
Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptNov 20 2020, 5:02 PM
Dzahn triaged this task as Low priority.Nov 20 2020, 5:04 PM
Bstorm added a subscriber: Bstorm.Nov 20 2020, 10:04 PM

That actually makes sense since that path was removed, and I bet it was coded in somewhere.

Change 642558 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] nfs monitoring: remove the old paws path

https://gerrit.wikimedia.org/r/642558

Change 642558 merged by Bstorm:
[operations/puppet@production] nfs monitoring: remove the old paws path

https://gerrit.wikimedia.org/r/642558

Thanks a lot for the quick response!

lmata moved this task from Backlog to Radar on the SRE board.Nov 23 2020, 4:19 PM
lmata moved this task from Inbox to Radar on the observability board.
Dzahn closed this task as Resolved.Nov 24 2020, 2:01 AM
Dzahn claimed this task.

I did not get the mail today anymore. Thank you Brooke!