The Product Safety and Integrity team is testing a bot detection service (hCaptcha) as a possible replacement for the current CAPTCHA system.
In a deployment planned for Q2 (2025 - 2026), the team will release hCaptcha in 99.9% passive mode on several wikis, and in 100% passive mode [i] on enwiki/frwiki in the source/wikitext editor [ii] and move to a 99.9% passive mode after one week of data collection
This task involves the work of:
- Editing + Product Analytics:
- Document the metrics we need to be gathered as part of this trial so that we can evaluate its impact on the core editing funnel
- Review instrumentation plan (@EMill-WMF to link) being added as part of this trial and ensure it accords with established patterns in EditAttemptStep and VisualEditorFeatureUse. E.g. T409701.
- Product Safety and Integrity
- Confirm key editing metrics (see below) are implemented in a way that will enable Editing to evaluate
Open questions
- 1. At what point in the editing funnel will requests be sent off to hCaptcha? Knowing this will enable the Editing Team to determine the point from which to look at the metrics listed below.
- The hcaptcha.request() method is invoked when the user presses "Publish changes" in the wikitext editor. This introduces a period of latency while hCaptcha responds with a token, or in rare cases (~0.1% of sessions), a challenge. After the challenge is completed, a token is provided and the form submission continues.
Editing key metrics
WARNING: this section will not be considered final and ready for review by Product Safety and Integrity until @MNeisler has singed off on its completeness and accuracy.
| Learning objective | Metric(s) | Notes |
|---|---|---|
| Edit completion: What proportion of people go on to publish an edit after reaching the point in the editing funnel when a request is sent off to hCaptcha? | Use existing EditAttemptStep data. All users without skipcaptcha right will have hcaptcha.execute() evaluated as part of form submission. The VisualEditorFeatureUse event for hcaptcha_execute will make this more explicit in the funnel (T410146). | |
| Time to save: How much time does hCaptcha introduce into the publishing process? | We are tracking hcaptcha.execute() p10, p50, p90 in the Grafana dashboard and will have this segmented for edit interactions (T410008) | |
| Edit abandonment: What proportion of people abandon edits after reaching the point in the editing funnel when a request is sent off to hCaptcha? | The event generated in T410146: hCaptcha: Generate VisualEditorFeatureUse event when hCaptcha execute is invoked will allow for tracking this alongside existing data in EditAttemptStep | |
| No-JS users: What % of edits are blocked from publishing b/c someone is attempting to edit without JS enabled? | The data from rEWED7c1b4b8e13fd: EventLogging: Fix wikitext editor interface detection shows that approximately 0.6% of sessions from users without skipcaptcha are editing without JS | |
| Edit quality: How does the rate at which edits are reverted or cause an AbuseFilter to become activated vary between edits included as part of this trial and those that are not? | Measuring AbuseFilter trips is possible with the saveFailure event. We can track whether these drop for wikis with hCaptcha enabled. For revert rate, we can compare how things change between the week of 100% passive data on enwiki and frwiki and the subsequent shift to 99.9% passive | |
| [nice to have] Editor retention: How does editor retention vary among people whose edits are sent off to hCaptcha and those that are not? | This involves looking at retention rates for users who don't have skipcaptcha right, which is all new user accounts | |
i. Where "100% passive mode" means all edit included in the trial:
- Will be submitted to hCaptcha client for evaluation
- Will NOT be presented with a puzzle/challenge
ii. A trial of hCaptcha within visualeditor-based editing interfaces will occur at a later time.