Change 772032 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] Mediawiki appservers-https check: temporary tweak

https://gerrit.wikimedia.org/r/772032

Change 772032 merged by Volans:

[operations/puppet@production] Mediawiki appservers-https check: temporary tweak

https://gerrit.wikimedia.org/r/772032

As for:

Evaluate whether there's a better monitoring strategy for future expiries than paging on Sunday :)

I totally agree. Ideally, in my opinion:

• The availability check (that runs every minute) should not be coupled with the TLS certificate expiration check.
• The availability check should surely ensure that the TLS connection works fine, but the check for certificare expiration could be decoupled and be run at a much lower frequency (something between every hour and once a day).
• The certificate expiration check should ideally page only during working hours.

As for:

Identify whether there are any more about to expire, for which we weren't alerted

I've done a quick check and seems that in the next 30 days (2592000 seconds) we get just those and the rendering ones (are they still used? A quick git grep in the puppet repo comes back empty handed)

root@puppetmaster1001:~# for file in $(ls /var/lib/puppet/server/ssl/ca/signed/*.pem); do openssl x509 -checkend 2592000 -noout -in "${file}" &> /dev/null; if [[ "${?}" -ne "0" ]]; then echo -n "${file}: "; openssl x509 -enddate -noout -in "${file}"; fi; done
/var/lib/puppet/server/ssl/ca/signed/api.svc.codfw.wmnet.pem: notAfter=Mar 28 10:22:42 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/api.svc.eqiad.wmnet.pem: notAfter=Mar 28 14:39:16 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/appservers.svc.codfw.wmnet.pem: notAfter=Mar 28 11:05:14 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/appservers.svc.eqiad.wmnet.pem: notAfter=Mar 28 16:54:40 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/rendering.svc.codfw.wmnet.pem: notAfter=Mar 28 08:30:54 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/rendering.svc.eqiad.wmnet.pem: notAfter=Mar 28 13:56:30 2022 GMT
root@puppetmaster1001:~#

Change 772034 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] Mediawiki api-https check: temporary tweak

https://gerrit.wikimedia.org/r/772034

As for:

We didn't get a matching page, but the cert for api.svc.[site].wmnet will also expire Monday, 2022-03-28 14:39:16.

This seems to be cause by the checking URL setup in hieradata/common/service.yaml (/w/api.php?action=query&meta=siteinfo).
This screenshot from icinga shows that the check is indeed critical but is not in CRITICAL status for Icinga, that means that it didn't return 2 as exit code. I hence think that the URL is not properly quoted/encoded and the check goes in background because of the &, making it return 0 and then capturing the output anyway.

I'm sending a patch to fix this one too.

Change 772034 merged by Volans:

[operations/puppet@production] Mediawiki api-https check: temporary tweak

https://gerrit.wikimedia.org/r/772034

And now both checks for api are in warning, as they should be, added item to the task description.

I wholeheartedly agree with the points made here, I'll add that as part of this quarter's work on the monitoring section of service::catalog (T291946) I'll be switching the alertmanager alerts to paging (perhaps in conjuction with the legacy ones for a couple of weeks).

In other words we'll get:

• decoupled availability and certificate checks (i.e. https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-sre/probes.yaml)
• less exceptions and corner cases in configuration (e.g. one-line configuration to add a standard probe, two if a custom path is desired)
• more frequent availability checking (and tunable, from icinga's 60s to 15s now)
• probes are run from prometheus hosts within the same site (i.e. the availability checks for services don't also test the reachability eqiad <-> site where the service is hosted)

Change 772790 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Renew certificates for appservers and apiservers

https://gerrit.wikimedia.org/r/772790

Change 772790 merged by JMeybohm:

[operations/puppet@production] Renew certificates for appservers and apiservers

https://gerrit.wikimedia.org/r/772790

Change 772792 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Renew certificates for appservers and apiservers

https://gerrit.wikimedia.org/r/772792

Change 772792 merged by JMeybohm:

[operations/puppet@production] Renew certificates for appservers and apiservers

https://gerrit.wikimedia.org/r/772792

Certs have been renewed (with cergen managed ones). Thanks @Joe for pairing!

Thanks! I think we can now destroy the ones in the Puppet CA mentioned in T304237#7790839 at this point.

In T304237#7795994, @Volans wrote:

Thanks! I think we can now destroy the ones in the Puppet CA mentioned in T304237#7790839 at this point.

Would that be puppet cert clean <CN>?

In T304237#7797398, @JMeybohm wrote:

In T304237#7795994, @Volans wrote:

Thanks! I think we can now destroy the ones in the Puppet CA mentioned in T304237#7790839 at this point.

Would that be puppet cert clean <CN>?

That might work too, Spicerack uses puppet ca --disable_warnings deprecations destroy {hostname}.

In T304237#7797420, @Volans wrote:

In T304237#7797398, @JMeybohm wrote:

In T304237#7795994, @Volans wrote:

Thanks! I think we can now destroy the ones in the Puppet CA mentioned in T304237#7790839 at this point.

Would that be puppet cert clean <CN>?

That might work too, Spicerack uses puppet ca --disable_warnings deprecations destroy {hostname}.

either of theses should work, ping me if you need a hand.

In T304237#7790839, @Volans wrote:

root@puppetmaster1001:~# for file in $(ls /var/lib/puppet/server/ssl/ca/signed/*.pem); do openssl x509 -checkend 2592000 -noout -in "${file}" &> /dev/null; if [[ "${?}" -ne "0" ]]; then echo -n "${file}: "; openssl x509 -enddate -noout -in "${file}"; fi; done
/var/lib/puppet/server/ssl/ca/signed/api.svc.codfw.wmnet.pem: notAfter=Mar 28 10:22:42 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/api.svc.eqiad.wmnet.pem: notAfter=Mar 28 14:39:16 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/appservers.svc.codfw.wmnet.pem: notAfter=Mar 28 11:05:14 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/appservers.svc.eqiad.wmnet.pem: notAfter=Mar 28 16:54:40 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/rendering.svc.codfw.wmnet.pem: notAfter=Mar 28 08:30:54 2022 GMT
/var/lib/puppet/server/ssl/ca/signed/rendering.svc.eqiad.wmnet.pem: notAfter=Mar 28 13:56:30 2022 GMT
root@puppetmaster1001:~#

Those certs have now been removed. Thanks @jbond

Change 774825 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Restore check_https_url command for api/appservers

https://gerrit.wikimedia.org/r/774825

Change 774825 merged by Filippo Giunchedi:

[operations/puppet@production] Restore check_https_url command for api/appservers

https://gerrit.wikimedia.org/r/774825

@lmata is it still valid?