I noticed the probes for librenms.wikimedia.org were failing on prometheus hosts with "unable to get local issuer" or "x509: certificate signed by unknown authority"; starting about 10 days ago.
As far as I can tell apache is sending LE R10 intermediate (i.e. RSA) though presenting a EC certificate (thus issued by E6)
$ openssl s_client -showcerts -connect librenms.wikimedia.org:443 CONNECTED(00000003) depth=0 CN = librenms.wikimedia.org verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = librenms.wikimedia.org verify error:num=21:unable to verify the first certificate verify return:1 depth=0 CN = librenms.wikimedia.org verify return:1 --- Certificate chain 0 s:CN = librenms.wikimedia.org i:C = US, O = Let's Encrypt, CN = E6 -----BEGIN CERTIFICATE----- MIIDijCCAxGgAwIBAgISA82mxKwFcPVopSRpYErgo3XdMAoGCCqGSM49BAMDMDIx ... 2PBa82ZQ950IjJNWyAt8/sAAbvj84LZPEGI5l9sDxHENZgQlI4LrnltfKskCMDdw 9Q+nLakzuXri0gIda39+Li6RgXmXCqYOm+/UxwoE/A3rsC2pYJaJ1LuPGQygSw== -----END CERTIFICATE----- 1 s:C = US, O = Let's Encrypt, CN = R10 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 -----BEGIN CERTIFICATE----- MIIFBTCCAu2gAwIBAgIQS6hSk/eaL6JzBkuoBI110DANBgkqhkiG9w0BAQsFADBP ... KPpdzvvtTnOPlC7SQZSYmdunr3Bf9b77AiC/ZidstK36dRILKz7OA54= -----END CERTIFICATE----- --- Server certificate subject=CN = librenms.wikimedia.org issuer=C = US, O = Let's Encrypt, CN = E6
I found a simple fix in the form of just not configuring rsa certificates in apache for librenms, which given it is only an internal service that's fine wrt compatibility.
With only EC certs of course X1 + E6 chain is sent correctly:
$ openssl s_client -showcerts -connect librenms.wikimedia.org:443 CONNECTED(00000003) depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = E6 verify return:1 depth=0 CN = librenms.wikimedia.org verify return:1 --- Certificate chain 0 s:CN = librenms.wikimedia.org i:C = US, O = Let's Encrypt, CN = E6 -----BEGIN CERTIFICATE----- MIIDijCCAxGgAwIBAgISA82mxKwFcPVopSRpYErgo3XdMAoGCCqGSM49BAMDMDIx ... 9Q+nLakzuXri0gIda39+Li6RgXmXCqYOm+/UxwoE/A3rsC2pYJaJ1LuPGQygSw== -----END CERTIFICATE----- 1 s:C = US, O = Let's Encrypt, CN = E6 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 -----BEGIN CERTIFICATE----- MIIEVzCCAj+gAwIBAgIRALBXPpFzlydw27SHyzpFKzgwDQYJKoZIhvcNAQELBQAw ... EwOy59Hdm0PT/Er/84dDV0CSjdR/2XuZM3kpysSKLgD1cKiDA+IRguODCxfO9cyY Ig46v9mFmBvyH04= -----END CERTIFICATE----- --- Server certificate subject=CN = librenms.wikimedia.org issuer=C = US, O = Let's Encrypt, CN = E6