Page MenuHomePhabricator

Grafana reports ALL docker mounts in a spammy way
Closed, ResolvedPublic


When a container starts, Docker mount a filesystem under /var/lib/docker/devicemmapper/mnt/. Docker creates these on the fly and they are all rather short lived.

Diamond eventually collect the disk space usages on those and report to Graphite.

Shinken monitors the graphite data and it ends up emitting an alarm because the metrics are outdated (no valid datapoints found).

image.png (939×1 px, 375 KB)

They can be seen at

  1. These entries for the docker hosts will be filling up graphite I expect! a new metric or 2 will be added for every job run
  2. It makes the dashboard for disk usage pretty unusable.

There is a similar issue in production with the Icinga check_disk probe T178454.

Event Timeline

For T1075: Audit groups of metrics in Graphite that allocate a lot of disk space @fgiunchedi did a tweak to disable reporting diskI/O from partitions:

diamond::collector { 'DiskSpace':
    settings => {
        filesystems => 'ext2,ext3,ext4,xfs,fuse.fuse_dfs,fat32,fat16,btrfs',

I guess the DiskSpace collector can be tweaked in such a way? It has:


filesystems to examine
Default: 'ext2, ext3, ext4, xfs, glusterfs, nfs, nfs4, ntfs, hfs, fat32, fat16, btrfs'


A list of regex patterns. Any filesystem matching any of these patterns will be excluded from disk space metrics collection.

exclude_filters = ^/boot, ^/mnt

Default: '^/export/home'

So I guess we could exclude .*docker/devicemapper/.*

That sounds like a good solution.

Change 393215 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] diamond: skip DiskSpace for Docker containers

Change 393215 merged by Alexandros Kosiaris:
[operations/puppet@production] diamond: skip DiskSpace for Docker containers

hashar claimed this task.
hashar closed this task as Resolved.