hCaptcha: Self-host secure-api.js code in /static directory
Closed, ResolvedPublic
Actions

Description

Summary

The contents of secure-api.js can be self-hosted. We should consider doing this for improved auditability, especially if/when the contents change.

Specification

secure-api.js should be loaded from a versioned repository. The endpoint, assethost, and other query parameters should continue to be set to values that use the proxy.

Technical notes

TBD

Acceptance criteria

secure-api.js is stored in a publicly versioned repository (https://gitlab.wikimedia.org/repos/product-safety-and-integrity/hcaptcha-secure-api-vendor)
secure-api.js is loaded directly or indirectly via the publicly versioned repository
After loading our self-hosted secure-api.js, requests to hCaptcha from the client continue to route through the proxy

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
hCaptcha: Roll out self-hosted secure-api.js to all wikis	Kosta Harlan	operations/mediawiki-config	master	+2 -9
Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"	Kosta Harlan	operations/mediawiki-config	master	+3 -10
Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"	Kosta Harlan	operations/mediawiki-config	master	+10 -3
Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis"	Kosta Harlan	operations/mediawiki-config	master	+3 -10
hCaptcha: Load self-hosted secure-api.js on group0 wikis	Kosta Harlan	operations/mediawiki-config	master	+10 -3
scap.cfg.erb: Add hcaptcha checkout in production	Ahmon Dancy	operations/puppet	production	+1 -1
.gitignore: Add /static/hcaptcha/	Ahmon Dancy	operations/mediawiki-config	master	+2 -0
hCaptcha: Use ResourceLoader for locally-hosted secure-api.js	Kosta Harlan	mediawiki/extensions/ConfirmEdit	master	+20 -9

Customize query in gerrit

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
Serve secure-api.js under a "1/" version path segment	repos/product-safety-and-integrity/hcaptcha-secure-api-vendor!3	kharlan	fix-self-hosted-path-version-segment	main
Pin hCaptcha secure-api.js at be2fb915	repos/product-safety-and-integrity/hcaptcha-secure-api-vendor!2	kharlan	pin-secure-api-be2fb915	main
scap prep: Add foss_violations support	repos/releng/scap!1186	dancy	master-I6f40ef7727c44839b8038831824bc4d29b722e06	master

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
					Restricted Task
		Resolved		kostajh	T403829 hCaptcha: Self-host secure-api.js code in /static directory

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Reedy moved this task from Backlog to Feature Requests/Improvements on the ConfirmEdit (CAPTCHA extension) board.Sep 12 2025, 4:42 PM

kostajh moved this task from Backlog to In progress on the Bot detection and mitigation (WE4.2 hCaptcha account creation trial) board.Sep 16 2025, 7:47 AM

For licensing reasons, as discussed in 1185314 (and to my knowledge as well), we are unable to host any proprietary code to our repos. I am afraid, same principle applies to our puppet repo as well.

We don't have a clear path forwards on this right now. T404251: hCaptcha: Enable version pinning and subresource integrity is implemented, however.

Change #1185314 abandoned by Kosta Harlan:

[mediawiki/extensions/ConfirmEdit@master] hCaptcha: Use ResourceLoader for locally-hosted secure-api.js

Reason:

Abandoning this for now

https://gerrit.wikimedia.org/r/1185314

Maintenance_bot removed a project: Patch-For-Review.Sep 18 2025, 12:30 PM

kostajh changed the task status from Open to Stalled.Sep 25 2025, 1:41 PM

kostajh removed kostajh as the assignee of this task.

kostajh mentioned this in T418477: Improve resilience of hCaptcha API URL loading to transient network issues.Feb 26 2026, 1:46 PM

kostajh mentioned this in T421464: hCaptcha: Stop using urldownloader for health checks of the secure-api.js file.Mar 30 2026, 9:34 AM

Bringing this back into our sprint to reassess and evaluate options.

One idea mentioned in this thread is to create a new repo which just has secure-api.js in it, get that deployed, and have foreign-resources.yaml point to it. I don't know if there would be similar licensing issues with getting this new repo deployed, though.

In T403829#11847387, @kostajh wrote:

One idea mentioned in this thread is to create a new repo which just has secure-api.js in it, get that deployed, and have foreign-resources.yaml point to it. I don't know if there would be similar licensing issues with getting this new repo deployed, though.

Here's my proposal, based on what @Tgr suggested in the patch discussion:

Repo at https://gitlab.wikimedia.org/repos/product-safety-and-integrity/hcaptcha-secure-api-vendor, which contains the secure-api.js code. Every time we want to update the pinned version of secure-api.js, we generate new directories in there (there's an update script in that repo to facilitate this)
Add a submodule in operations/mediawiki-config at docroot/mediawiki/static/hcaptcha. scap will then materialize it at deploy time. (Other submodules exist in operations/mediawiki-config that pull from Gerrit repos, so this would be a first for using a GitLab repo, but I imagine that is OK.)
Update $wgHCaptchaApiUrl to point to $wgServer . '/static/hcaptcha/<hash>/secure-api.js'

Proposed .gitmodules entry for mediawiki-config:

[submodule "docroot/mediawiki/static/hcaptcha"]
       path = docroot/mediawiki/static/hcaptcha
       url = https://gitlab.wikimedia.org/repos/product-safety-and-integrity/hcaptcha-secure-api-vendor

Clients would then download secure-api.js from e.g. https://en.wikipedia.org/static/hcaptcha/{hash}/secure-api.js. Files served from static/ have a long cache control (1 year, from operations/puppet in puppet/modules/mediawiki/files/apache/modules/expires.conf) and should be behind the CDN.

With this approach, we don't need any puppet or apache vhost changes, and we don't add any proprietary or non-licensed code directly in operations/mediawiki-config, operations/puppet or in ConfirmEdit repos. And we will reduce an extra step in requests that clients currently need (client -> WMF proxy -> hCaptcha CDN) to just client -> WMF CDN.

@ssingh @thcipriani could you please review this proposal from Traffic and Release-Engineering-Team point of view? cc also @Dreamy_Jazz

kostajh changed the task status from Stalled to Open.Apr 23 2026, 7:52 AM

kostajh edited projects, added: Product Safety and Integrity (Sprint Tulip (Apr 13 - May 1)); removed: Product Safety and Integrity.

• hector.arroyo subscribed.Apr 23 2026, 8:24 AM

Dreamy_Jazz awarded a token.Apr 23 2026, 8:32 AM

kostajh added a parent task: Restricted Task.Apr 23 2026, 8:44 AM

kostajh moved this task from Backlog to In progress on the Product Safety and Integrity (Sprint Tulip (Apr 13 - May 1)) board.Apr 23 2026, 11:48 AM

A_smart_kitten subscribed.Apr 23 2026, 3:34 PM

Overall seems unproblematic, two notes.

Adding a submodule to mediawiki-config is similar to how portals is deployed. Portals is singular, but it works ok.

Couple things to note:

Hosting this on GitLab would mean that MediaWiki deployments are dependent on GitLab.
- Concretely, scap backport (if this change were made today) would fail if GitLab were down or otherwise unavailable to the deploy host.
- GitLab is already load-bearing -- currently, deploying a service would be difficult if GitLab were down. So the material change is making it load-bearing for MediaWiki.
- CC, @LSobanski as my SRE, GitLab counter-part there in case he has thoughts.
IIRC, there are some cache purge scripts that are run when a new version of portals rolls out (sync-portals script)
- I'll leave that detail to traffic since folks are CC'd.

GitLab is already load-bearing -- currently, deploying a service would be difficult if GitLab were down. So the material change is making it load-bearing for MediaWiki.

Let's say GitLab is having a bad day on a day we'd like to deploy. How much work would it be (and could it be done by SRE alone without much coordination) to cut the tie that day and deploy MediaWiki anyway?

In T403829#11860957, @EMill-WMF wrote:

GitLab is already load-bearing -- currently, deploying a service would be difficult if GitLab were down. So the material change is making it load-bearing for MediaWiki.

Let's say GitLab is having a bad day on a day we'd like to deploy. How much work would it be (and could it be done by SRE alone without much coordination) to cut the tie that day and deploy MediaWiki anyway?

It could be done by anyone with deployment shell access in a few steps via ssh. The one-step methods we have would fail, but deploys could be done with a scap sync-file vs. scap backport or spiderpig.

The plan @kostajh outlined is OK by me.

In T403829#11852721, @thcipriani wrote:

IIRC, there are some cache purge scripts that are run when a new version of portals rolls out (sync-portals script)

I'll leave that detail to traffic since folks are CC'd.

In our case, we shouldn't need to purge any caches, since the new version of secure-api.js would be in a new directory that uses the secure-api.js version, e.g. 73f27c192b38c05ce2ebce596a0e28f88a2a56bf/secure-api.js

Adding a submodule to mediawiki-config would mean that repo would be deployed to deployment-prep, where the WMCS Terms of Use prohibits non-free software.

In T403829#11866418, @taavi wrote:

Adding a submodule to mediawiki-config would mean that repo would be deployed to deployment-prep, where the WMCS Terms of Use prohibits non-free software.

It sounds like deployment-prep would need to not clone that particular submodule. If you have an alternative proposal, please let us know.

@thcipriani and I will try to propose some alternatives.

OKryva-WMF edited projects, added: Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)); removed: Product Safety and Integrity (Sprint Tulip (Apr 13 - May 1)).May 1 2026, 7:45 AM

OKryva-WMF moved this task from Backlog to In progress on the Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)) board.May 1 2026, 7:45 AM

In T403829#11852721, @thcipriani wrote:

Overall seems unproblematic, two notes.

Adding a submodule to mediawiki-config is similar to how portals is deployed. Portals is singular, but it works ok.

Couple things to note:

Hosting this on GitLab would mean that MediaWiki deployments are dependent on GitLab.

Concretely, scap backport (if this change were made today) would fail if GitLab were down or otherwise unavailable to the deploy host.

GitLab is already load-bearing -- currently, deploying a service would be difficult if GitLab were down. So the material change is making it load-bearing for MediaWiki.

CC, @LSobanski as my SRE, GitLab counter-part there in case he has thoughts.

IIRC, there are some cache purge scripts that are run when a new version of portals rolls out (sync-portals script)

I'll leave that detail to traffic since folks are CC'd.

GitLab undergoes regular patching (usually once every two weeks). The downtime for these is sub 5 minutes. Based on what I read above this should not be a problem as long as it doesn't overlap with Mediawiki deployments, which should not be the case as our maintenance window is on Fridays.

I would like to point out there is https://config-master.wikimedia.org/ which is already a web server that hosts files related to Mediawiki config / deployment.

This used to be on the deployment server itself afair, but nowadays these are 2 dedicated VMs.

There should also already be a mechanism that automatically gets files from git repos to that webserver.

Just putting this out there as a possible option that has existed a long time.

In T403829#11890732, @Dzahn wrote:

I would like to point out there is https://config-master.wikimedia.org/ which is already a web server that hosts files related to Mediawiki config / deployment.

This used to be on the deployment server itself afair, but nowadays these are 2 dedicated VMs.

There should also already be a mechanism that automatically gets files from git repos to that webserver.

Just putting this out there as a possible option that has existed a long time.

Thank you @Dzahn, that is helpful to know. I think we have a plan that will work, I will ask @thcipriani to update this task with the proposal.

/srv/mediawiki-staging is a checkout of operations/mediawiki-config. The checkout is kept up-to-date by scap in both beta and production.

Proposal, for getting it to the deploy server and into production images, high-level:

Scap checks out hcaptcha-secure-api-vendor to /srv/mediawiki-staging/static/hcaptcha as part of scap prep
- Scap's config is aware of the cluster it runs in, so this can be configured to only happen in production in /etc/scap.cfg (puppet:scap.cfg.erb)
operations/mediawiki-config: would need to gitignore that path, since it is now managed outside that repository: .gitignore static/hcaptcha
file is available at <webroot>/w/static/hcaptcha/<sha>/secure-api.js

CDanis subscribed.May 11 2026, 4:23 PM

kostajh mentioned this in T426178: hCaptcha: Failed to load resource: Origin (domain) is not allowed by Access-Control-Allow-Origin.May 13 2026, 1:59 PM

dancy updated https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1186

scap prep: Add foss_violations support

Change #1287007 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap.cfg.erb: Add hcaptcha checkout in production

https://gerrit.wikimedia.org/r/1287007

gerritbot added a project: Patch-For-Review.May 13 2026, 7:30 PM

Change #1287026 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/mediawiki-config@master] .gitignore: Add /static/hcaptcha/

https://gerrit.wikimedia.org/r/1287026

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1186

scap prep: Add foss_violations support

Change #1287026 merged by jenkins-bot:

[operations/mediawiki-config@master] .gitignore: Add /static/hcaptcha/

https://gerrit.wikimedia.org/r/1287026

Mentioned in SAL (#wikimedia-operations) [2026-05-18T13:46:03Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1287895|Store uncomputed references delta as null, not 0 (T426002)]], [[gerrit:1287026|.gitignore: Add /static/hcaptcha/ (T403829)]]

Mentioned in SAL (#wikimedia-operations) [2026-05-18T13:47:51Z] <lucaswerkmeister-wmde@deploy1003> daimona, lucaswerkmeister-wmde, dancy: Backport for [[gerrit:1287895|Store uncomputed references delta as null, not 0 (T426002)]], [[gerrit:1287026|.gitignore: Add /static/hcaptcha/ (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-05-18T13:56:00Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1287895|Store uncomputed references delta as null, not 0 (T426002)]], [[gerrit:1287026|.gitignore: Add /static/hcaptcha/ (T403829)]] (duration: 09m 57s)

kostajh renamed this task from hCaptcha: Self-host secure-api.js code to hCaptcha: Self-host secure-api.js code in /static directory.May 20 2026, 8:17 PM

OKryva-WMF edited projects, added: Product Safety and Integrity (Sprint Iris (May 25 - Jun 12)); removed: Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)).May 25 2026, 12:41 PM

OKryva-WMF moved this task from Backlog to In progress on the Product Safety and Integrity (Sprint Iris (May 25 - Jun 12)) board.May 25 2026, 12:42 PM

Dzahn unsubscribed.May 26 2026, 9:37 PM

kharlan opened https://gitlab.wikimedia.org/repos/product-safety-and-integrity/hcaptcha-secure-api-vendor/-/merge_requests/2

Pin hCaptcha secure-api.js at be2fb915

Change #1295909 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] hCaptcha: Load self-hosted secure-api.js on group0 wikis

https://gerrit.wikimedia.org/r/1295909

Change #1295910 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] hCaptcha: Roll out self-hosted secure-api.js to all wikis

https://gerrit.wikimedia.org/r/1295910

Change #1287007 merged by Ssingh:

[operations/puppet@production] scap.cfg.erb: Add hcaptcha checkout in production

https://gerrit.wikimedia.org/r/1287007

kharlan merged https://gitlab.wikimedia.org/repos/product-safety-and-integrity/hcaptcha-secure-api-vendor/-/merge_requests/2

Pin hCaptcha secure-api.js at be2fb915

Change #1295909 merged by jenkins-bot:

[operations/mediawiki-config@master] hCaptcha: Load self-hosted secure-api.js on group0 wikis

https://gerrit.wikimedia.org/r/1295909

Mentioned in SAL (#wikimedia-operations) [2026-06-02T15:49:51Z] <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1295909|hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-02T15:51:39Z] <kharlan@deploy1003> kharlan: Backport for [[gerrit:1295909|hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Change #1296624 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis"

https://gerrit.wikimedia.org/r/1296624

Mentioned in SAL (#wikimedia-operations) [2026-06-02T15:59:40Z] <kharlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1295909|hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s)

Change #1296624 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis"

https://gerrit.wikimedia.org/r/1296624

Mentioned in SAL (#wikimedia-operations) [2026-06-02T16:03:20Z] <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1296624|Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-02T16:05:09Z] <kharlan@deploy1003> kharlan: Backport for [[gerrit:1296624|Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-06-02T16:10:01Z] <kharlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1296624|Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s)

kharlan opened https://gitlab.wikimedia.org/repos/product-safety-and-integrity/hcaptcha-secure-api-vendor/-/merge_requests/3

Serve secure-api.js under a "1/" version path segment

Change #1296635 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"

https://gerrit.wikimedia.org/r/1296635

kharlan merged https://gitlab.wikimedia.org/repos/product-safety-and-integrity/hcaptcha-secure-api-vendor/-/merge_requests/3

Serve secure-api.js under a "1/" version path segment

Change #1296635 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"

https://gerrit.wikimedia.org/r/1296635

Mentioned in SAL (#wikimedia-operations) [2026-06-03T08:41:22Z] <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1296635|Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-03T08:47:23Z] <kharlan@deploy1003> kharlan: Backport for [[gerrit:1296635|Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Change #1297064 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"

https://gerrit.wikimedia.org/r/1297064

Mentioned in SAL (#wikimedia-operations) [2026-06-03T08:53:05Z] <kharlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1296635|Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s)

Change #1297064 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"

https://gerrit.wikimedia.org/r/1297064

Mentioned in SAL (#wikimedia-operations) [2026-06-03T08:55:04Z] <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1297064|Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-03T08:59:02Z] <kharlan@deploy1003> kharlan: Backport for [[gerrit:1297064|Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-06-03T09:05:59Z] <kharlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1297064|Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s)

Change #1295910 merged by jenkins-bot:

[operations/mediawiki-config@master] hCaptcha: Roll out self-hosted secure-api.js to all wikis

https://gerrit.wikimedia.org/r/1295910

Mentioned in SAL (#wikimedia-operations) [2026-06-03T13:20:33Z] <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1295910|hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]]

kostajh edited projects, added: Bot detection and mitigation (WE4.10 hCaptcha); removed: Bot detection and mitigation (WE4.2 hCaptcha editing trial).Jun 3 2026, 1:50 PM

@dom_walden I think this one could skip QA, but I will leave it for you to review.

kostajh moved this task from In progress to QA in Prod on the Product Safety and Integrity (Sprint Iris (May 25 - Jun 12)) board.Jun 3 2026, 1:53 PM

kostajh updated the task description. (Show Details)

OKryva-WMF edited projects, added: Product Safety and Integrity (Sprint Rose (Jun 15 - Jun 26)); removed: Product Safety and Integrity (Sprint Iris (May 25 - Jun 12)).Jun 15 2026, 2:32 PM

OKryva-WMF moved this task from Backlog to QA in Prod on the Product Safety and Integrity (Sprint Rose (Jun 15 - Jun 26)) board.Jun 15 2026, 2:32 PM

In T403829#11981364, @kostajh wrote:

@dom_walden I think this one could skip QA, but I will leave it for you to review.

Thanks. I will skip.

kostajh closed this task as Resolved.Jun 17 2026, 1:32 PM

Krinkle mentioned this in T429175: Scaling urldownloaders by adding redundancy and load balancing.Thu, Jul 30, 8:02 PM

	kostajh
	Sep 5 2025, 2:28 PM

hCaptcha: Self-host secure-api.js code in /static directoryClosed, ResolvedPublicActions