Page MenuHomePhabricator

hCaptcha: Enable A/B test for jawiki and zhwiki
Closed, ResolvedPublic

Description

Summary

jawiki and zhwiki both have FancyCaptcha enabled on edit and create actions for users without skipcaptcha right. This makes these wikis better suited for an A/B test with hCaptcha's 99.9% passive mode. In this task, we'll set up an A/B test (similar to T405239) where 90% of edge unique traffic gets hCaptcha 99.9% passive mode, and 10% of traffic gets FancyCaptcha.

Technical notes

We will need to create an event in the visualEditorFeatureUse event logging stream to indicate whether a given editing session ID is in the experiment or control group. For example:

const experiment = mw.xLab.getExperiment( 'fy25-26-we-4-2-hcaptcha-editing' );
mw.track( 'visualEditorFeatureUse', { feature: 'hcaptcha_experiment', action: experiment.isAssignedGroup( 'control', 'control-2') ? 'FancyCaptcha' : 'hCaptcha' } );

and that could be triggered via mw.trackSubscribe('editAttemptStep'), firing on the loaded event (h/t @DLynch)

Experiment in MPIC: https://mpic.wikimedia.org/experiment/fy25-26-we-4-2-hcaptcha-editing

Acceptance criteria

  • 90% of edge unique traffic gets hCaptcha in 99.9% passive mode on jawiki/zhwiki, 10% gets FancyCaptcha on edit/create actions
  • VEFU is updated to have an event to track which experiment group a user is in

Event Timeline

Change #1206808 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/WikimediaEvents@master] (WIP) hCaptcha: Record A/B test experiment group

https://gerrit.wikimedia.org/r/1206808

Change #1206830 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] hCaptcha: Enable A/B edit test on zhwiki and jawiki

https://gerrit.wikimedia.org/r/1206830

Change #1206808 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] hCaptcha: Record A/B test experiment group

https://gerrit.wikimedia.org/r/1206808

Change #1206960 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.3] hCaptcha: Record A/B test experiment group

https://gerrit.wikimedia.org/r/1206960

Change #1207108 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.2] hCaptcha: Record A/B test experiment group

https://gerrit.wikimedia.org/r/1207108

Change #1207108 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.2] hCaptcha: Record A/B test experiment group

https://gerrit.wikimedia.org/r/1207108

Change #1206960 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.3] hCaptcha: Record A/B test experiment group

https://gerrit.wikimedia.org/r/1206960

Mentioned in SAL (#wikimedia-operations) [2025-11-19T21:13:03Z] <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1207108|hCaptcha: Record A/B test experiment group (T410354)]], [[gerrit:1206960|hCaptcha: Record A/B test experiment group (T410354)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-19T21:17:47Z] <kharlan@deploy2002> kharlan: Backport for [[gerrit:1207108|hCaptcha: Record A/B test experiment group (T410354)]], [[gerrit:1206960|hCaptcha: Record A/B test experiment group (T410354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-11-19T21:24:19Z] <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1207108|hCaptcha: Record A/B test experiment group (T410354)]], [[gerrit:1206960|hCaptcha: Record A/B test experiment group (T410354)]] (duration: 11m 16s)

Change #1206830 merged by jenkins-bot:

[operations/mediawiki-config@master] hCaptcha: Enable A/B edit test on zhwiki and jawiki

https://gerrit.wikimedia.org/r/1206830

Mentioned in SAL (#wikimedia-operations) [2025-11-19T21:42:16Z] <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1206830|hCaptcha: Enable A/B edit test on zhwiki and jawiki (T410354)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-19T21:47:01Z] <kharlan@deploy2002> kharlan: Backport for [[gerrit:1206830|hCaptcha: Enable A/B edit test on zhwiki and jawiki (T410354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-11-19T21:53:11Z] <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1206830|hCaptcha: Enable A/B edit test on zhwiki and jawiki (T410354)]] (duration: 10m 55s)

Nothing to test?

It would be helpful to validate that we are seeing the expected 90/10 split of hCaptcha/FancyCaptcha, by looking at VisualEditorFeatureUse events from https://gerrit.wikimedia.org/r/c/1206808

Change #1210621 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] hCaptcha: Adjust addurl config for zhwiki and jawiki

https://gerrit.wikimedia.org/r/1210621

Change #1210621 merged by jenkins-bot:

[operations/mediawiki-config@master] hCaptcha: Adjust addurl config for zhwiki and jawiki

https://gerrit.wikimedia.org/r/1210621

Mentioned in SAL (#wikimedia-operations) [2025-11-25T08:28:06Z] <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1210621|hCaptcha: Adjust addurl config for zhwiki and jawiki (T410354 T409957)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-25T08:32:37Z] <kharlan@deploy2002> kharlan: Backport for [[gerrit:1210621|hCaptcha: Adjust addurl config for zhwiki and jawiki (T410354 T409957)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-11-25T08:41:55Z] <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1210621|hCaptcha: Adjust addurl config for zhwiki and jawiki (T410354 T409957)]] (duration: 13m 49s)

Change #1238072 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] Remove A/B test for hCaptcha editing

https://gerrit.wikimedia.org/r/1238072

Change #1238072 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove A/B test for hCaptcha editing

https://gerrit.wikimedia.org/r/1238072

Mentioned in SAL (#wikimedia-operations) [2026-02-11T21:27:53Z] <cjming@deploy2002> Started scap sync-world: Backport for [[gerrit:1238072|Remove A/B test for hCaptcha editing (T410354)]]

Mentioned in SAL (#wikimedia-operations) [2026-02-11T21:30:15Z] <cjming@deploy2002> kharlan, cjming: Backport for [[gerrit:1238072|Remove A/B test for hCaptcha editing (T410354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-02-11T21:34:47Z] <cjming@deploy2002> Finished scap sync-world: Backport for [[gerrit:1238072|Remove A/B test for hCaptcha editing (T410354)]] (duration: 06m 53s)