DX App Synthetic Monitoring App - watchmouse alert flapping due to CA expiration
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• lmata
	Oct 6 2021, 4:34 AM

Project Tags

Referenced Files

None

Subscribers

Description

Several checks in external monitoring / watchmouse have been flapping since Sept 30th due to Let's Encrypt's Root Certificate expiring as referred to in T283164: Let's Encrypt issuance chains update

It seems there are several hosts within the watchmouse probe infrastructure with stale CAs.

sample message below:

According to monitor 'Phabricator', the https service on
'phabricator.wikimedia.org' has not been working as specified since 2021-10-06
04:11:42 GMT (UTC +00:00).

Message: SSL certificate problem: certificate has expired (Peer certificate
cannot be authenticated with given CA certificates)

Affected checks (possibly incomplete):

Phabricator
https services - foundationwiki
https services - wikiquote
https services - wiktionary
https services - wikivoyage
https services - wikibooks
https services - wikisource
https services - loginwiki
https services - mediawiki
https services - wikidata
Wiki platform [[w:de:Main Page]] (s5)
Static assets (CSS/JS

Will track resolution of the monitoring issue below.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	prometheus: add blackbox generic "watchrat" http/s static check support	Herron	operations/puppet	production	+61 -16

Customize query in gerrit

Related Objects

Mentioned In: T281812: Audit/Assess meta monitoring strategy
T299147: Retire WatchMouse (CA DX APP)
Mentioned Here: T299147: Retire WatchMouse (CA DX APP)
T97099: Squeeze value out of external monitoring services
T283164: Let's Encrypt issuance chains update

Event Timeline

• lmata created this task.Oct 6 2021, 4:34 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 6 2021, 4:34 AM

• lmata changed the task status from Open to In Progress.Oct 6 2021, 4:34 AM

• lmata claimed this task.

• lmata triaged this task as Medium priority.

• lmata moved this task from Inbox to In progress on the SRE Observability (FY2021/2022-Q2) board.

• lmata updated the task description. (Show Details)

There doesn't seem to be many immediate available options: I think we can temporarily disable SSL certificate checking. I have posted a question/thread in the Broadcom/DX monitoring community to reach out and see if there is a workaround or fix on the Broadcom side.

BTullis subscribed.Oct 6 2021, 8:32 AM

worth mentioning that the checks are configured to "remind" every hour. However the flapping comes as their network doesn't seem to be evenly patched for CAs, some probes fail, and some work as intended causing the "flapping" and excess reminders.

I've been learning that as checks rotate through different probes, some will resolve to esams/eqsin and receive a non-LetsEncrypt cert that has no issues, however when probes hit eqiad/codfw/ulsfo the LetsEncrypt certificate is served with the issue of the expired CA cert (thanks @Volans). Sharing it here for context.

I have spoken to a few folks internally and have arrived at the bold conclusion to disable SSL certificate validation on the DX App Mon/Watchmouse on the following HTTPS checks:

https services - wiktionary
https services - commons
https services - foundationwiki
https services - loginwiki
https services - mediawiki
https services - wikibooks
https services - wikidata
https services - wikinews
https services - wikipedia
https services - wikiquote
https services - wikisource
https services - wikiversity
https services - wikivoyage
Phabricator
icinga-https
donate https

This will mitigate the noise problem for now. There are a few HTTP checks that do redirect to HTTPS and trigger the expired LE CA alert. These HTTP checks cannot be mitigated as there is no "ignore SSL" or similar option.

I will keep this open to continue to keep track of the error and see if this gets resolved on the DX App/WM side within a couple of weeks.

This fix isn't optimal so please update this task if you have an alternate or better solution than disabling SSL validation for HTTPS checks. Meanwhile, this experience will feed into T97099: Squeeze value out of external monitoring services and similar tasks about formulating a better future state for external monitoring

@lmata any news from their side? We're still getting the frequent emails for the remaining checks that keep flapping.

Friendly reminder as those email are still coming ;)

I've been getting them only for shop.wikimedia.org though. A LOT OF THEM.

yes currently the SSL ones are disabled and the flappy ones are the HTTP ones that redirect to HTTPS (and then fail) I'm trying to track down who owns these alerts and find a suitable internal check. next quarter we will be addressing ext-mon as a wholistic project. task soon to be linked here

The problem I see with the prolonging of this issue is that all recipients of those emails are most likely by now trained to ignore all alerts from this specific external monitoring or have setup rules to ignore them in their inbox.
As a result real alerts might get ignored, like the one we got this morning for ALERT! mr1-ulsfo OOB: Packets lost (100%).

@lmata almost another month has passed and we're approaching the holiday season, is there by any chance any news from their side?

Volans added a subscriber: faidon.Dec 6 2021, 1:45 PM

MatthewVernon subscribed.Dec 6 2021, 1:48 PM

@Volans there are no news yet, I have reached out again and am now working on a replacement and will turning off these alerts in favor for an internal icinga check. We have plans in Q3 to address external monitoring as a whole as part of our OKR.

herron subscribed.Dec 13 2021, 5:01 PM

Change 747550 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] prometheus: add blackbox http/s static check support

https://gerrit.wikimedia.org/r/747550

gerritbot added a project: Patch-For-Review.Dec 15 2021, 5:37 PM

herron moved this task from FY2021/2022-Q2 to FY2021/2022-Q3 on the SRE Observability board.Jan 11 2022, 7:23 PM

herron edited projects, added: SRE Observability (FY2021/2022-Q3); removed: SRE Observability (FY2021/2022-Q2).

• lmata mentioned this in T299147: Retire WatchMouse (CA DX APP).Jan 13 2022, 5:11 PM

• lmata moved this task from Inbox to Up next on the SRE Observability (FY2021/2022-Q3) board.Jan 21 2022, 8:09 PM

Change 747550 merged by Herron:

[operations/puppet@production] prometheus: add blackbox generic \"watchrat\" http/s static check support

https://gerrit.wikimedia.org/r/747550