When a container starts, Docker mount a filesystem under /var/lib/docker/devicemmapper/mnt/. Docker creates these on the fly and they are all rather short lived.
Diamond eventually collect the disk space usages on those and report to Graphite.
Shinken monitors the graphite data and it ends up emitting an alarm because the metrics are outdated (no valid datapoints found).
They can be seen at https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?panelId=18&fullscreen&orgId=1&var-project=integration&var-server=integration-slave-docker-1001&var-server=integration-slave-docker-1002&var-server=integration-slave-docker-1003&var-server=integration-slave-docker-1004&from=now-90d&to=now
- These entries for the docker hosts will be filling up graphite I expect! a new metric or 2 will be added for every job run
- It makes the dashboard for disk usage pretty unusable.
There is a similar issue in production with the Icinga check_disk probe T178454.