Page MenuHomePhabricator

tools-workers: prometheus-node-exporter `Error on statfs() system call for... permission denied`
Closed, ResolvedPublic

Description

This is spamming the logs of all tools-worker-* nodes:

Dec 12 06:35:08 tools-worker-1015 prometheus-node-exporter[654]: time="2018-12-12T06:35:08Z" level=error msg="Error on statfs() system call for "/run/docker/netns/47128f4653d2": permission denied" source="filesystem_linux.go:57"
Dec 12 06:35:08 tools-worker-1015 prometheus-node-exporter[654]: time="2018-12-12T06:35:08Z" level=error msg="Error on statfs() system call for "/var/lib/docker/devicemapper/mnt/7f729ce826fa165cd4ee4e55125680e54ee6630f8413c2d78d6f05c28852b16f": permission denied" source="filesystem_linux.go:57"

Similar to what is described in T184469 but maybe with different paths / fstypes:

  • /var/lib/docker/devicemapper/mnt
  • /var/lib/docker/containers
  • /run/docker/netns
  • /var/lib/kubelet/pods/

Event Timeline

Change 479424 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] prometheus-node-exporter: Ignore Docker/Kubelet mount points

https://gerrit.wikimedia.org/r/479424

GTirloni triaged this task as Low priority.
GTirloni moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 479424 merged by GTirloni:
[operations/puppet@production] prometheus-node-exporter: Ignore Docker/Kubelet mount points

https://gerrit.wikimedia.org/r/479424

Mentioned in SAL (#wikimedia-operations) [2018-12-17T18:29:37Z] <gtirloni> prometheus-node-exporter: Ignore Docker/Kubelet mount points T211810