hCaptcha: Adjust tolerance of intermittent network errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	kostajh
	Feb 9 2026, 5:59 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

dom_walden

kostajh

Description

Summary

In T404204: Investigate options for automatic fallback to FancyCAPTCHA, we implemented a failover mode to load FancyCaptcha in the event that hCaptcha is unreachable. For reasons that are still being investigated, there are intermittent connection errors to load secure-api.js for outbound network calls (see https://logstash.wikimedia.org/goto/3638725468dcb59bc3e917335776afa1 for other examples). We currently set "failover mode" if secure-api.js is unreachable. We should instead tolerate a certain number of errors per period of time.

Acceptance criteria

the health checker implementation is reworked to accept a threshold of errors per window of time

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	ConfirmEdit: Add error count threshold for apiUrl health checks	Kosta Harlan	mediawiki/extensions/ConfirmEdit	wmf/1.46.0-wmf.14	+141 -22
	ConfirmEdit: Add error count threshold for apiUrl health checks	Kosta Harlan	mediawiki/extensions/ConfirmEdit	master	+141 -22

Customize query in gerrit

Related Objects

Mentioned In: T418477: Improve resilience of hCaptcha API URL loading to transient network issues
Mentioned Here: T404204: Investigate options for automatic fallback to FancyCAPTCHA

Event Timeline

kostajh created this task.Feb 9 2026, 5:59 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2026, 5:59 AM

Change #1237732 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ConfirmEdit@master] ConfirmEdit: Add error count threshold for apiUrl health checks

https://gerrit.wikimedia.org/r/1237732

gerritbot added a project: Patch-For-Review.Feb 9 2026, 6:12 AM

Change #1237732 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@master] ConfirmEdit: Add error count threshold for apiUrl health checks

https://gerrit.wikimedia.org/r/1237732

Maintenance_bot removed a project: Patch-For-Review.Feb 9 2026, 9:31 AM

kostajh edited projects, added: Product Safety and Integrity (Sprint Daffodil (Jan 19 - Feb 6)); removed: Product Safety and Integrity.Feb 9 2026, 9:36 AM

kostajh moved this task from Backlog to Needs QA on the Product Safety and Integrity (Sprint Daffodil (Jan 19 - Feb 6)) board.

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.15; 2026-02-10).Feb 9 2026, 10:00 AM

OKryva-WMF edited projects, added: Product Safety and Integrity (Sprint Flower (Feb 9 - Feb 27)); removed: Product Safety and Integrity (Sprint Daffodil (Jan 19 - Feb 6)).Feb 10 2026, 8:28 AM

OKryva-WMF moved this task from Backlog to Needs QA on the Product Safety and Integrity (Sprint Flower (Feb 9 - Feb 27)) board.Feb 10 2026, 8:29 AM

Change #1238436 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ConfirmEdit@wmf/1.46.0-wmf.14] ConfirmEdit: Add error count threshold for apiUrl health checks

https://gerrit.wikimedia.org/r/1238436

gerritbot added a project: Patch-For-Review.Feb 10 2026, 10:11 PM

Change #1238436 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@wmf/1.46.0-wmf.14] ConfirmEdit: Add error count threshold for apiUrl health checks

https://gerrit.wikimedia.org/r/1238436

Mentioned in SAL (#wikimedia-operations) [2026-02-10T22:28:06Z] <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1238436|ConfirmEdit: Add error count threshold for apiUrl health checks (T416817)]]

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2026, 10:30 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-10T22:32:20Z] <kharlan@deploy2002> kharlan: Backport for [[gerrit:1238436|ConfirmEdit: Add error count threshold for apiUrl health checks (T416817)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-02-10T22:40:40Z] <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1238436|ConfirmEdit: Add error count threshold for apiUrl health checks (T416817)]] (duration: 12m 34s)

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.14; 2026-02-03); removed: MW-1.46-notes (1.46.0-wmf.15; 2026-02-10).Feb 10 2026, 11:00 PM

OKryva-WMF edited projects, added: Product Safety and Integrity (Sprint Crocus (Mar 2 - Mar 20)); removed: Product Safety and Integrity (Sprint Flower (Feb 9 - Feb 27)).Mar 2 2026, 4:18 PM

OKryva-WMF moved this task from Backlog to Needs QA on the Product Safety and Integrity (Sprint Crocus (Mar 2 - Mar 20)) board.Mar 2 2026, 4:19 PM

OKryva-WMF edited projects, added: Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))); removed: Product Safety and Integrity (Sprint Crocus (Mar 2 - Mar 20)).Mar 23 2026, 3:56 PM

OKryva-WMF moved this task from Backlog to Needs QA on the Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))) board.

Testing locally and setting $wgHCaptchaApiUrl = 'https://localhost' to simulate the hCaptcha service being down (note: I don't know if this is a realistic simulation).

When someone opens the edit page or Special:CreateAccount, we enter into the objectcache two variables: confirmedit-hcaptcha-available and confirmedit-hcaptcha-apiurl-error-count.

Service	`confirmedit-hcaptcha-available`	`confirmedit-hcaptcha-apiurl-error-count`	Failover?
Down	true, expiry=30s	1, expiry=30m	No
Down (third try)	false, expiry=10m	3+, expiry=unchanged	Yes
Down (`...available` has expired)	false, expiry=10m	3+, expiry=unchanged	Yes
Up (`...available` has expired)	true, expiry=1m	removed	No
Down (`...available` and `...error-count` has expired)	true, expiry=30s	1, expiry=30m	No

Notice in the last row, if the hCaptcha service is down for a prolonged period, every 30 minutes (when ...error-count expires) we leave failover mode for a couple of minutes while we wait for 3 failures to access the hCaptcha service.

dom_walden mentioned this in T418477: Improve resilience of hCaptcha API URL loading to transient network issues.Mar 27 2026, 8:44 AM

kostajh closed this task as Resolved.Mar 27 2026, 9:10 AM

kostajh claimed this task.

• hector.arroyo moved this task from Backlog to Done on the Bot detection and mitigation (WE4.2 hCaptcha editing trial) board.Mar 27 2026, 10:37 AM

hCaptcha: Adjust tolerance of intermittent network errorsClosed, ResolvedPublicActions