The Product Safety and Integrity team is testing a bot detection service (hCaptcha) as a possible replacement for the current CAPTCHA system.
In a deployment planned for Q2 (2025 - 2026), the team will release hCaptcha in 99.9% passive mode on several wikis, and in 100% passive mode [i] on enwiki/frwiki in the source/wikitext editor [ii] and move to a 99.9% passive mode after one week of data collection
This task involves the work of:
# **Editing + Product Analytics**:
## Document the metrics we need to be gathered as part of this trial so that we can evaluate its impact on the core editing funnel
## Review instrumentation plan !!(@EMill-WMF to link)!! being added as part of this trial and ensure it accords with established patterns in EditAttemptStep and VisualEditorFeatureUse. //E.g. T409701.//
# **Product Safety and Integrity**
## Confirm key editing metrics (see below) are implemented in a way that will enable Editing to evaluate
=== Open questions
- [x] 1. At what point in the editing funnel will requests be sent off to hCaptcha? //Knowing this will enable the Editing Team to determine the point from which to look at the metrics listed below.//
- The `hcaptcha.request()` method is invoked when the user presses "Publish changes" in the wikitext editor. This introduces a period of latency while hCaptcha responds with a token, or in rare cases (~0.1% of sessions), a challenge. After the challenge is completed, a token is provided and the form submission continues.
=== Editing key metrics
WARNING: this section will not be considered final and ready for review by Product Safety and Integrity until @MNeisler has singed off on its completeness and accuracy.
|Learning objective|Metric(s)|Notes
|---|---|---
|**Edit completion**: What proportion of people go on to publish an edit after reaching the point in the editing funnel when a request is sent off to hCaptcha?| Use existing EditAttemptStep data. All users without `skipcaptcha` right will have `hcaptcha.execute()` evaluated as part of form submission. The VisualEditorFeatureUse event for `hcaptcha_execute` will make this more explicit in the funnel (T410146). |
|**Time to save**: How much time does hCaptcha introduce into the publishing process?| We are tracking `hcaptcha.execute()` p10, p50, p90 in the [[ https://grafana-rw.wikimedia.org/d/441b2def-52e9-49d6-acad-91f5bb748989/hcaptcha-reverse-proxy-proxoid?orgId=1&from=now-7d&to=now&timezone=utc&var-instance=$__all&var-site=eqiad&var-wiki=ptwiki&var-wiki=jawiki&var-wiki=idwiki&var-wiki=zhwiki&var-wiki=trwiki&var-wiki=fawiki&var-wiki=frwiki&var-wiki=enwiki | Grafana dashboard ]] and will have this segmented for edit interactions (T410008) |
|**Edit abandonment:** What proportion of people abandon edits after reaching the point in the editing funnel when a request is sent off to hCaptcha?| The event generated in {T410146} will allow for tracking this alongside existing data in EditAttemptStep |
|**No-JS users:** What % of edits are blocked from publishing b/c someone is attempting to edit without JS enabled?| The data from {7c1b4b8e13fd32c624aa896ee58b19abe8c6e53f} shows that approximately 0.6% of sessions from users without `skipcaptcha` are editing without JS |
|**Edit quality:** How does the rate at which edits are reverted or cause an AbuseFilter to become activated vary between edits included as part of this trial and those that are not?| Measuring AbuseFilter trips is possible with the `saveFailure` event. We can track whether these drop for wikis with hCaptcha enabled. For revert rate, we can compare how things change between the week of 100% passive data on enwiki and frwiki and the subsequent shift to 99.9% passive |
|[nice to have] **Editor retention:** How does editor retention vary among people whose edits are sent off to hCaptcha and those that are not?| This involves looking at retention rates for users who don't have `skipcaptcha` right, which is all new user accounts |
---
i. Where "100% passive mode" means all edit included in the trial:
-- Will be submitted to hCaptcha client for evaluation
-- Will NOT be presented with a puzzle/challenge
ii. A trial of hCaptcha within visualeditor-based editing interfaces will occur at a later time.