additional issue is these alerts are flapping and keep coming back so a simple ACK wasn't enough. downtiming for a day for now

same thing happened today for lists.wikimedia.org, it alerted and then recovered 2 minutes later. In general we have renewal = 7 days and alerting = 7 days.

we should change them all to something like "6 days and 23 hours"

this sounds like https://etbe.coker.com.au/2021/10/20/strange-apache-reload-issue/ which I read yesterday via Planet Debian

In T293826#7445653, @Legoktm wrote:

this sounds like https://etbe.coker.com.au/2021/10/20/strange-apache-reload-issue/ which I read yesterday via Planet Debian

Interesting. Lists runs apache, although the cloudelastic hosts look like nginx.

I'm seeing 6 days being logged as well now. I wonder if the 7 days monitoring is coincidental, since that's the threshold we pass to check_http for check_https_lvs_on_port.

Oct 20 19:06:09 alert1001 icinga: SERVICE ALERT: cloudelastic.wikimedia.org;WMF Cloud -Omega Cluster- - Public Internet Port - HTTPS;CRITICAL;SOFT;2;CRITICAL - Certificate 'cloudelastic.wikimedia.org' expires in 6 day(s) (Wed 27 Oct 2021 07:00:23 PM GMT +0000)

Both the apache and nginx on cloudelastic and lists have been running for months, so fully restarting them seems worth a shot to see if that stops the flapping (for a few weeks/months anyway). I haven't touched them yet in case someone wants to review them in their current state first.

lists.wm.o is serving both the old and new cert, just like the blog post mentioned:

Mentioned in SAL (#wikimedia-operations) [2021-10-21T04:04:02Z] <legoktm> restarted apache on lists1001 so it only uses new TLS cert (T293826)

After restarting apache, and ~50 requests later, I'm only getting the new certificate. Marking as high because the monitoring is picking up a real issue, it's just flappy since ~half the requests pass and half fail.

Looking in syslog doesn't show much interesting AFAICT, except it seems like puppet/acme-chief is reloading apache every 2-3 days (not sure if that's intentional):

syslog.log-20211009.gz:Oct  8 10:23:33 lists1001 puppet-agent[32311]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211011.gz:Oct 10 22:22:46 lists1001 puppet-agent[22789]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211014.gz:Oct 13 10:23:26 lists1001 puppet-agent[8325]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211016.gz:Oct 15 22:23:11 lists1001 puppet-agent[11417]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211019.gz:Oct 18 11:23:03 lists1001 puppet-agent[30781]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]

(doesn't actually matter, seems like apache reloads config at midnight every day anyways)

Mentioned in SAL (#wikimedia-operations) [2021-10-21T06:35:55Z] <elukey> systemctl reload nginx on cloudelastic100[5,6] to pick up the new TLS certificate and clear alerts - T293826

Change 732611 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] tlsproxy::localssl: acme_chief should notify nginx

https://gerrit.wikimedia.org/r/732611

Maybe it is totally off, but I saw that the cloudelastic nodes use the tlsproxy::localssl define (via elasticsearch::tlsproxy) and it doesn't reload nginx when a new cert is available. Created https://gerrit.wikimedia.org/r/c/operations/puppet/+/732611 as proposal to improve the current situation, I see the same pattern applied to other acme_chief::cert usages.

Change 732611 merged by Elukey:

[operations/puppet@production] tlsproxy::localssl: acme_chief should notify nginx

https://gerrit.wikimedia.org/r/732611

The nginx problems should be fixed with https://gerrit.wikimedia.org/r/732611 in theory (so elastic/cloudelastic/relforge/etc..).

In T293826#7446833, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2021-10-21T04:04:02Z] <legoktm> restarted apache on lists1001 so it only uses new TLS cert (T293826)

This one seems sneaky to reproduce, I am wondering if recent versions of httpd have it fixed (upstream is now at 2.4.51, and several changes to mpm-event happened since .38, see https://www.apachelounge.com/Changelog-2.4.html and grep for event).

@Dzahn do you know if the list[12]* nodes are scheduled to be upgraded to Bullseye during the next few weeks? It would be really nice to test the new version of httpd, I think it should solve the issue that @Legoktm brought up.

@elukey Not list* but we could potentially test it with librenms.wikimedia.org. That fulfills the requirements of "uses LE", "has public IP" and "uses httpd" at least, unlike most other things.

Another option could be to use httpd from buster-backports, but https://packages.debian.org/buster-backports/apache2 still lists 2.4.48 (meanwhile on salsa I see .51, that contains a lot of CVE fixes from upstream). If the buster-backport version gets updated to .51 we could install it and avoid bullseye.

Mentioned in SAL (#wikimedia-cloud) [2021-12-06T13:23:49Z] <majavah> root@toolserver-proxy-01:~# systemctl restart apache2.service # working around T293826

Mentioned in SAL (#wikimedia-operations) [2021-12-19T17:10:32Z] <Amir1> restart apache2 on lists1001 (T293826)

that's intended, every time that acme-chief fetches fresh OCSP stapling responses it issues a reload of apache2

In T293826#7446839, @Legoktm wrote:

Looking in syslog doesn't show much interesting AFAICT, except it seems like puppet/acme-chief is reloading apache every 2-3 days (not sure if that's intentional):

syslog.log-20211009.gz:Oct  8 10:23:33 lists1001 puppet-agent[32311]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211011.gz:Oct 10 22:22:46 lists1001 puppet-agent[22789]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211014.gz:Oct 13 10:23:26 lists1001 puppet-agent[8325]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211016.gz:Oct 15 22:23:11 lists1001 puppet-agent[11417]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]
syslog.log-20211019.gz:Oct 18 11:23:03 lists1001 puppet-agent[30781]: (/etc/acmecerts/lists) Scheduling refresh of Service[apache2]

(doesn't actually matter, seems like apache reloads config at midnight every day anyways)

gerrit1001.wikimedia.org is showing up the issue:

In T308908#7957451, @hashar wrote:

From apache-status:

Current Time: Wednesday, 25-May-2022 15:16:44 UTC

Restart Time: Friday, 22-Apr-2022 19:59:53 UTC

+------------------------------------------------------------------------+
|    |     |        |Connections    |Threads  |Async connections         |
|Slot|PID  |Stopping|---------------+---------+--------------------------|
|    |     |        |total|accepting|busy|idle|writing|keep-alive|closing|
|----+-----+--------+-----+---------+----+----+-------+----------+-------|
|0   |6376 |no      |1    |yes      |0   |25  |0      |0         |1      |
|----+-----+--------+-----+---------+----+----+-------+----------+-------|
|1   |19045|no      |2    |yes      |1   |24  |0      |0         |1      |
|----+-----+--------+-----+---------+----+----+-------+----------+-------|
|2   |1390 |no (old |0    |yes      |0   |25  |0      |0         |0      |
|    |     |gen)    |     |         |    |    |       |          |       |
|----+-----+--------+-----+---------+----+----+-------+----------+-------|
|3   |6377 |no      |3    |yes      |0   |25  |0      |2         |0      |
|----+-----+--------+-----+---------+----+----+-------+----------+-------|
|Sum |4    |0       |6    |         |1   |99  |0      |2         |2      |
+------------------------------------------------------------------------+

The slot with pid 1390 is marked no (old gen) which is the one from April 23 in ps above.

We have Apache 2.4.38-3+deb10u7

I found upstream bug 63169: MPM event, stuck process after graceful: no (old gen) reported against 2.4.37 which says a graceful restart fails to kill some processes.

Looking at the stuck pid 1390:

apache2 1390 www-data   13w      REG       9,0    3119423    396450 /var/log/apache2/gerrit.wikimedia.org.https.error.log.1 (deleted)
apache2 1390 www-data   14w      REG       9,0       4215    396446 /var/log/apache2/gerrit.wikimedia.org.http.error.log.1 (deleted)
apache2 1390 www-data   15w      REG       9,0    2661276    396470 /var/log/apache2/other_vhosts_access.log.1 (deleted)
apache2 1390 www-data   16w      REG       9,0    7803259    396467 /var/log/apache2/other_vhosts_access-json.log.1 (deleted)
apache2 1390 www-data   17w      REG       9,0 9308490812    396448 /var/log/apache2/gerrit.wikimedia.org.https.access.log.1 (deleted)
apache2 1390 www-data   18w      REG       9,0    1172054    396445 /var/log/apache2/gerrit.wikimedia.org.http.access.log.1 (deleted)

The merge patch is https://svn.apache.org/viewvc?view=revision&revision=1893109 and is in Apache 2.4.49. That might be the one :]

The brute workaround is to restart Apache2 entirely?

This just alerted again:

14:09:04 <+icinga-wm> PROBLEM - mailman list info ssl expiry on lists1001 is CRITICAL: CRITICAL - Certificate lists.wikimedia.org expires in 7 day(s) (Sat 25 Jun 2022 07:55:09 AM GMT +0000). https://wikitech.wikimedia.org/wiki/Mailman/Monitoring

This happened again for lists1001. Requested (and it has been) restart in #wikimedia-sre