Page MenuHomePhabricator
Create Task
Maniphest T373369

Service puppetmaster1001:8141 has failed probes (http_puppetmaster1003_eqiad_wmnet_backend_https_ip4)
Closed, ResolvedPublic
Actions

Description

There is an alert related to puppetmaster1001's 8141 port:

Service puppetmaster1003:8141 has failed probes (http_puppetmaster1003_eqiad_wmnet_backend_https_ip4)

Following the Alert's logs links there is some clue:

target=https://[10.64.16.36]:8141/puppet/v3 msg="Error for HTTP request" err="Get \"https://10.64.16.36:8141/puppet/v3\": x509: certificate relies on legacy Common Name field, use SANs instead"

It seems coming from prometheus1006 mostly, that has 3 days of uptime (matching the start of the alerts).

In puppetmaster::monitoring we have the definition of the HTTP blackbox exporter, and I don't think that it is possible to change puppetmaster1001's TLS cert to have SANs. Running the exporter with GODEBUG="x509ignoreCN=0" should fix the issue, but from a quick glance this is not supported yet in Puppet.

Related Objects

Event Timeline

elukey created this task.Aug 26 2024, 4:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2024, 4:56 PM
elukey renamed this task from Service puppetmaster1003:8141 has failed probes (http_puppetmaster1003_eqiad_wmnet_backend_https_ip4) to Service puppetmaster1001:8141 has failed probes (http_puppetmaster1003_eqiad_wmnet_backend_https_ip4).Aug 26 2024, 4:56 PM
fgiunchedi added a comment.Aug 27 2024, 7:47 AM

Thank you for filing the task, I was also looking at the same failed probes due to the recent Prometheus Bookworm upgrade. tl;dr is I think it is safe to ack these alerts and I will do so, see also https://phabricator.wikimedia.org/T326657#10090776

Stashbot added a comment.Aug 27 2024, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2024-08-27T07:50:22Z] <godog> ack probedown for puppetmaster:8181 - T373369

fgiunchedi closed this task as Resolved.Aug 27 2024, 7:55 AM
fgiunchedi claimed this task.

I'm tentatively resolving since the silences are in place, please feel free to reopen as needed!

fgiunchedi mentioned this in T326657: Add prometheus-https load balancer.Aug 27 2024, 11:10 AM
Scott_French mentioned this in T393727: Prometheus black-box probes for all puppetmaster hosts are failing.May 8 2025, 6:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits