So because something spammed the logs in 2018 due to permissions.. asking about this in 2023 for a valid use case is declined within minutes?

I closed this task because as far as I can tell that previous task from a few years back was about this exact same issue and nothing in this task suggests that something has changed since this issue was last investigated. If you have something that suggests that something has changed, feel free to add that, but for now I don't see anything actionable here that justifies keeping the task open.

@thcipriani @dancy T327060 was needed for buildkit to make automatic builds work for production which needed T327435 which needed this ticket which is declined as you can see above. this blocks efforts from your sprint week

In T328972#8591345, @taavi wrote:>

nothing in this task suggests that something has changed since this issue was last investigated.

This ticket was asking to change it, which you declined with the reason that it has not changed.

I don't see anything actionable here that justifies keeping the task open.

The action would be to stop filtering /var/lib/docker which is why I created the ticket. It was declined without any hint of discussion within 5 minutes.

Is there a local puppetmaster in the gitlab-runners project where someone could try removing the ban on /var/lib/docker from https://gerrit.wikimedia.org/r/c/operations/puppet/+/479424/2/modules/prometheus/manifests/node_exporter.pp#35 to see what errors if any come out today?

Yes, there is a local puppetmaster. gitlab-runners-puppetmaster-01.

I just made an edit to /etc/puppet/modules/prometheus/manifests/node_exporter line 34 to remove /var/lib/docker from the $ignored_mount_points.

Thanks for the hint @bd808 !

Metrics for /var/lib/docker are available in devtools project now.

runner-1029:~$ curl 172.16.1.135:9100/metrics | grep "node_filesystem_avail_bytes"
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda",fstype="ext4",mountpoint="/var/lib/docker"} 3.0610907136e+10
...

Also the dashboard for Shared Runners contains /var/lib/docker now:
https://grafana-cloud.wikimedia.org/d/FrErwP0Vk/gitlab-runner-overview?orgId=1&from=now-12h&to=now

In T328972#8592421, @Jelto wrote:

Metrics for /var/lib/docker are available in devtools project now.

Someone should check back after a few full days of operation, but at the moment fgrep prometheus-node-exporter /var/log/*.log on runner-1029.gitlab-runners.eqiad1.wikimedia.cloud is not showing file system read permission spam as previously seen in T211810: tools-workers: prometheus-node-exporter `Error on statfs() system call for... permission denied`.

Wow, well, I expected either it would work OR it would spam again, but not both at the same time. So it gets "permission denied" but still does get the data we want.. is that odd?

In T328972#8594820, @Dzahn wrote:

Wow, well, I expected either it would work OR it would spam again, but not both at the same time. So it gets "permission denied" but still does get the data we want.. is that odd?

I missed typing "not" in my comment which unfortunately inverted the intent of the update. :) This has now been corrected.

oh:) that's good news! cool, thank you

Change 888009 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] prometheus::node_exporter: remove /var/lib/docker from ignored_mount_points

https://gerrit.wikimedia.org/r/888009

Change 888009 merged by Jelto:

[operations/puppet@production] prometheus::node_exporter: remove /var/lib/docker from ignored_mount_points

https://gerrit.wikimedia.org/r/888009