Maniphest T406880

hCaptcha: Implement alerts
Closed, ResolvedPublic
Actions

Description

Summary

When ConfirmEdit declares that it is in "failover mode", we want to alert the Product Safety and Integrity and SRE.

Technical notes

We have a warning level message in the captcha log channel for hCaptcha is unavailable, falling back to FancyCaptcha. When this fires, we should create an alert
We have a panel recording the request times for the HCaptchaEnterpriseHealthChecker::isAvailable() call in https://grafana-rw.wikimedia.org/d/441b2def-52e9-49d6-acad-91f5bb748989/hcaptcha-reverse-proxy-proxoid?orgId=1&from=now-3h&to=now&timezone=utc&var-instance=$__all&var-site=eqiad&var-wiki=ptwiki&var-wiki=jawiki&var-wiki=idwiki&var-wiki=zhwiki&var-wiki=trwiki&var-wiki=fawiki&var-wiki=frwiki&viewPanel=panel-36 . If this drops to 0, the service can be considered to have failed and we should generate an alert

The alerts can go to:

#psi-alerts channel in Slack
a team in SRE (TBD)

Acceptance criteria

Alert is defined for the Logstash entry for hCaptcha is unavailable, falling back to FancyCaptcha
Alert is defined for when the isAvailable() Grafana panel drops to 0

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	HCaptchaEnterpriseHealthChecker: enhance isAvailable tracking	mediawiki/extensions/ConfirmEdit	master	+14 -7

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		kostajh	T404204 Investigate options for automatic fallback to FancyCAPTCHA
		Resolved		colewhite	T406880 hCaptcha: Implement alerts

Event Timeline

kostajh created this task.Oct 9 2025, 3:05 PM

Restricted Application added subscribers: Huji, Stang, Aklapper. · View Herald TranscriptOct 9 2025, 3:05 PM

colewhite claimed this task.Oct 10 2025, 10:48 PM

colewhite added a project: Observability-Logging.

Please have a look: https://grafana-rw.wikimedia.org/alerting/grafana/af0o85cnfuxhcb/view

kostajh moved this task from Priority backlog to In progress on the Product Safety and Integrity (Sprint Apfel Strudel (Sep 29 - Oct 17)) board.Oct 15 2025, 11:48 AM

In T406880#11266008, @colewhite wrote:

Please have a look: https://grafana-rw.wikimedia.org/alerting/grafana/af0o85cnfuxhcb/view

This looks good to me, thank you! @colewhite would you be able to also set up an alert for the Logstash entry for hCaptcha is unavailable, falling back to FancyCaptcha?

@ssingh do you have any requests for additional alerts based on the Grafana dashboard?

[Adding Raine @kamila as well.]

Additional thoughts:

IIUC, hcaptcha.execute() errors seem like another useful signal. It's rather catch-all, so it'd need to be routed to both dev and SRE, but seems good to have. Or would that overlap with the fallback?

As for latency alerts, those would likely be coming from hCaptcha, so they would likely not be actionable by us and thus I don't think they'd be valuable.

OKryva-WMF edited projects, added Product Safety and Integrity (Sprint Mint Choc Chip Ice Cream (Oct 20 - Nov 7)); removed Product Safety and Integrity (Sprint Apfel Strudel (Sep 29 - Oct 17)).Oct 20 2025, 2:25 PM

OKryva-WMF moved this task from Backlog to Needs review on the Product Safety and Integrity (Sprint Mint Choc Chip Ice Cream (Oct 20 - Nov 7)) board.Oct 20 2025, 2:26 PM

In T406880#11289372, @Raine wrote:

Additional thoughts:

IIUC, hcaptcha.execute() errors seem like another useful signal. It's rather catch-all, so it'd need to be routed to both dev and SRE, but seems good to have. Or would that overlap with the fallback?

As for latency alerts, those would likely be coming from hCaptcha, so they would likely not be actionable by us and thus I don't think they'd be valuable.

We could alert on hcaptcha.execute() errors, but I think we would have to set a pretty high threshold (e.g. > 100 in the last hour). Generally, a low number of these are expected.

In T406880#11298908, @kostajh wrote:

We could alert on hcaptcha.execute() errors, but I think we would have to set a pretty high threshold (e.g. > 100 in the last hour). Generally, a low number of these are expected.

You're right, and on second thought, I'm not sure it's a good idea. Giving the internet an unintentional Klaxon is probably bad :D

In T406880#11286467, @kostajh wrote:

@colewhite would you be able to also set up an alert for the Logstash entry for hCaptcha is unavailable, falling back to FancyCaptcha?

From the code, it looks like that log entry hinges on $services->getService( 'HCaptchaEnterpriseHealthChecker' )->isAvailable(): Since we have access to this code, I'd like to propose a change to the ConfirmEdit extension to back this alert.

Change #1198151 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/ConfirmEdit@master] HCaptchaEnterpriseHealthChecker: enhance isAvailable tracking

https://gerrit.wikimedia.org/r/1198151

gerritbot added a project: Patch-For-Review.Oct 22 2025, 8:27 PM

Change #1198151 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@master] HCaptchaEnterpriseHealthChecker: enhance isAvailable tracking

https://gerrit.wikimedia.org/r/1198151

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2025, 9:30 AM

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.25; 2025-10-28).Oct 24 2025, 10:00 AM

OKryva-WMF edited projects, added Product Safety and Integrity (Crepes au Chocolat (Sprint Nov 10 - Nov 28)); removed Product Safety and Integrity (Sprint Mint Choc Chip Ice Cream (Oct 20 - Nov 7)).Nov 10 2025, 12:14 PM

OKryva-WMF moved this task from Backlog to Needs review on the Product Safety and Integrity (Crepes au Chocolat (Sprint Nov 10 - Nov 28)) board.Nov 10 2025, 12:14 PM

OKryva-WMF edited projects, added Product Safety and Integrity (Sprint Mince Pie Dec 1 - Dec 12); removed Product Safety and Integrity (Crepes au Chocolat (Sprint Nov 10 - Nov 28)).Dec 1 2025, 9:05 AM

OKryva-WMF moved this task from Backlog to Needs review on the Product Safety and Integrity (Sprint Mince Pie Dec 1 - Dec 12) board.Dec 1 2025, 9:05 AM

It seems there is nothing to review here. Tentatively moving to Done for not knowing where else this should go

Restricted Application added a subscriber: Dragoniez. · View Herald TranscriptDec 3 2025, 3:36 PM

kostajh closed this task as Resolved.Dec 3 2025, 3:37 PM

Aklapper added a project: ConfirmEdit (CAPTCHA extension).Dec 4 2025, 11:58 AM

In T406880#11300166, @colewhite wrote:

In T406880#11286467, @kostajh wrote:

@colewhite would you be able to also set up an alert for the Logstash entry for hCaptcha is unavailable, falling back to FancyCaptcha?

From the code, it looks like that log entry hinges on $services->getService( 'HCaptchaEnterpriseHealthChecker' )->isAvailable(): Since we have access to this code, I'd like to propose a change to the ConfirmEdit extension to back this alert.

@colewhite it seems like an alert should have fired given these logs https://logstash.wikimedia.org/goto/6aee202d1a5f1feb5ac0504d4e2dbcd6 , but I do not see one in #psi-alerts.

In T406880#11435587, @kostajh wrote:

@colewhite it seems like an alert should have fired given these logs https://logstash.wikimedia.org/goto/6aee202d1a5f1feb5ac0504d4e2dbcd6 , but I do not see one in #psi-alerts.

You're right, an alert did not fire. From the previous alert definition, this hcaptcha failure was considered partial and the alert was tuned for a complete failure (rate <= 0) on the overall state of is_available. This was not taking into account the result label we added.

I've used this event to adjust based on a known failure condition. The alert now fires when: sum(increase(mediawiki_ConfirmEdit_hcaptcha_enterprise_health_checker_is_available_seconds_count{result="false"}[5m])) > 0

Playground

I'm going to optimistically mark this as resolved since the events are fairly clear in Prometheus now. Please do reach out if we need to adjust further.

Stang unsubscribed.Dec 8 2025, 9:33 PM

hCaptcha: Implement alertsClosed, ResolvedPublicActions