Based on these results, we will move ahead with deploying the banner to all users and probably execute the rollout in small batches.

Updated a dashboard in the Growthbook to monitor the health and progress for the experiment https://growthbook.wikimedia.org/experiment/exp_2423gnmp4cgdar#results

Successful Edits (hCaptcha wikis)
Look at the no-JS edits from non-autoconfirmed accounts. The data is for May 2026 when we have the autoconfirmed user group added in the event table.

For logins, we only have the UA information, so we are considering these patterns in UA text as grade C browsers.

Account Creation
According to this conversation, noJS account creations in event.mediawiki_product_metrics_special_create_account have

Successful edits

• I ran the following query for successful noJS edit, the rate seems extremely high. Looks like VisualEditorFeatureUse might not have all the editing sessions with JS-specific signals.

@GGalofre-WMF I added an additional 3 wikis in the previous comments.

Summary of metrics collected so far：

• Notebook https://gitlab.wikimedia.org/conniecc1/misc/-/blob/main/T414503_article_creation_guidance_baselines.ipynb
• Data is from 01/01/2026 to 02/28/2026
• Mobile VE edits
• Junior editors: We are defining junior editors as editors with less than 100 edits, as we have used in other tasks.
• Pilot wikis: arwiki, bnwiki, jawiki, ptwiki, fawiki, trwiki, simplewiki, eswiki, frwiki

We selected 4 similar-sized wikis where the hCaptcha trial is not active to compare the same user segment across these wikis and the trial wikis: eswiki, itwiki, plwiki, kowiki.

• Editing Completion & Abandon Rate：The data from non-trial Wikis establishes a completion rate of ~97% for the targeted user groups, and abandonment is <3%. Some of the trial wikis, like trwiki and ptwiki have a slightly lower completion rate. However, wiki idwiki, abandonment jumped to 32.5%. idwiki has a higher block rate on noJS edits; maybe that's the reason for the lower editing completion rate.
• No-JS Users Block Rate: The hCaptcha trial wikis have a higher no-JS edits block rate compared to wikis not in the trial, especially in idwiki and ptwiki.
• Revert rate: Across both groups, the targeted user segment is 6 to 10 times more likely to be reverted than those that are not.
• Abuse filter hit: In trial wikis, these users are 12.7 times more likely to trigger a filter than those that do not.

Editing Completion & Abandon Rate

We originally intended to perform a pre- and post-deployment comparison to measure the impact of hCaptcha. However, we identified a data quality issue in the event_sanitized.editattemptstep table: for historical sessions, the wiki column is NULL for desktop events where event.editor_interface = wikitext and event.integration = 'page'. We cannot filter or aggregate historical metrics for specific wikis.
As a result, this analysis uses data between 3/1/2026 and 4/1/2026 from the event table. The metrics presented reflect the current performance and behavior patterns.

In T422886#11823739, @Tchanders wrote:

In T422886#11823702, @A_smart_kitten wrote:

T422886#11807083

cchen changed the visibility from "cchen (connie)" to "All Users".

Side-note, I just want to check -- is there a reason for this task to only be visible to logged-in users on Phabricator, or can the visiblity be changed to Public (No Login Required)? /genq

I would have thought it could be public. @cchen what do you think?

Query for qualified users

User list for enwiki from 03/01/2026 - 04/01/2026

User pull for ptwiki and trwiki
https://docs.google.com/spreadsheets/d/1vEzfWxLrnrZnq98jnQl1a6FQ8L-faVuyHXfp6jgUYgo/edit?gid=0#gid=0

Query for qualified users

@Strainu the data processing here follows the previous thresholds analysis.

Below are the thresholds that provide 1% FPR for each wiki:

The growth trends (articles & editors) for Wikipedias that graduated from Incubator are in this sheet. https://docs.google.com/spreadsheets/d/1Inl71Z_1ycQf4rizJD8qLUPww2wSzJe7jXcQX9BPFP8/edit?gid=783438882#gid=783438882

Adding visualizations

The analysis is updated in the spreadsheet, we are looking at all active Wikipedia.

• Top 20 Gadgets, there are 217 Wikipedia are using these gadgets : https://docs.google.com/spreadsheets/d/1XVKUFTz5_PRwxpvdeNOzArnLjocbZ1ni2U820J8h1vg/edit?gid=0#gid=0
• Wikis that missing all top 20 gadgets: https://docs.google.com/spreadsheets/d/1XVKUFTz5_PRwxpvdeNOzArnLjocbZ1ni2U820J8h1vg/edit?gid=849565516#gid=849565516
• For top 20 Gadgets edits and users, we found the only field mentioning gadgets in edits data is editing comment. In this case, we are looking at December edits data in English Wikipedia. And some edits might be missing if the gadget names are not mentioned properly in the editing comments. https://docs.google.com/spreadsheets/d/1XVKUFTz5_PRwxpvdeNOzArnLjocbZ1ni2U820J8h1vg/edit?gid=1141128630#gid=1141128630

As part 2 of the evaluation, we collect the following data 3 months after the experiment:

Here is the analysis comparing baseline data with data post-change ( September 11th to December 1st). Please refer to this notebook for detailed analysis.

Final analysis in this report and entry_points_analysis

The first version of the dashboard in here https://superset.wikimedia.org/superset/dashboard/wikifunctions-ve-usage.

Tested and confirmed that all the events are logged in visualeditorfeatureuse schema.

I tested and confirmed the translation_difficulty_level logged in dashboard_translation_start events in mediawiki_content_translation_event table.

Please find Part 1 of the analysis here https://nbviewer.org/urls/gitlab.wikimedia.org/conniecc1/we2.1.1-kr-2025/-/raw/main/central-notification-banner-PART1.ipynb

Worked on:

• Collected and processed article-creation data for Wikipedia (2023–2025).
• Collected and processed content-translation event data for Wikipedia (2023–2025).
• Analyzed data for CX entry points and translation start step
  • Breakdown by user global edit bucket
  • Breakdown by the comparative wiki size of target languages
• Proportion of articles created by a user using CX (within 2023 - 2025), to check the user's preference for using CX vs. other tools.
  • Breakdown by user global edit bucket
  • Breakdown by the comparative wiki size of target languages
  • Breakdown by mobile vs. desktop
  • Also checked user tenures
• Proportion of user counts in each global edit bucket, to check when users start to use CX (with their editing experience change).
  • Breakdown by the comparative wiki size of target languages
  • Breakdown by mobile vs. desktop

Notebook for tracking baselines: https://gitlab.wikimedia.org/conniecc1/we2.1.2-kr-2025/-/blob/main/baselines-tracking.ipynb

@Nikerabbit, I confirm the publish_success events now contain valid published_revision_id and valid published_page_id.

Here are the baselines using data between April 1, 2025, and July 1, 2025. Please refer to this notebook for detailed calculations.

Thank you @JMeybohm!

@SBisson Hmmm, let's put 'unknown' then, in case there's any data issue and we need to check missing values and unknown values.

@SBisson, sure, to confirm the possible values for difficulty level will be easy, medium, and hard.

@SBisson, I’d recommend adding a new column to the schema to log the recommendation difficulty level.
And it can apply to event types dashboard_translation_start.

I queried the data destination, event.mediawiki_content_translation_event for August. There's 1 event with dashboard_open event type with event source as recent_edit.

I agree that using the global edit count bucket makes sense.

@kostajh please finds the reports for ip reputation data analysis related to blocked account https://analytics.wikimedia.org/published/reports/ip_reputation_data/report_blocked_accounts.html and related to reverted edits https://analytics.wikimedia.org/published/reports/ip_reputation_data/report_reverted_edits.html here.

The report is posted here https://analytics.wikimedia.org/published/reports/magru_datacenter_analysis/report.html
and related code is here https://gitlab.wikimedia.org/conniecc1/magru-data-cernter-analysis/-/tree/main?ref_type=heads

The queries files have been checked in through Gitlab merge, changes include:

@mforns - I've added you as a reviewer on this patch to add the mediawiki_product_metrics_incident_reporting_system_interaction schema to the allowlist. Here's a link to the instrumentation to fields. Please let me know if you have any questions or suggested revisions. Thank you!

In T383299#10528073, @kostajh wrote:

In T383299#10527033, @cchen wrote:

Here's the link to the IRS metrics dashboard. Please review it and let me know if anything needs to be updated.
https://superset.wikimedia.org/superset/dashboard/561

LGTM, thanks @cchen! Will the dashboard automatically include new projects as the incident reporting system is enabled on them?

Here's the link to the IRS metrics dashboard. Please review it and let me know if anything needs to be updated.
https://superset.wikimedia.org/superset/dashboard/561