On dismissal data:
We expected every dismissal to include one of four required reasons (Other, Irrelevant, Uncertain, or Common Knowledge), since users must select a reason when dismissing Reference Check. In practice, we see cases where Reference Check was shown and the user dismissed; 11% of dismissals have no reason provided.

@ppelberg: I’ve completed an analysis of Reference Check priority leading indicators as described in T405421.
See the summary of findings below; the full report includes expanded metrics and breakdowns by platform, number of checks, and user experience.
Note: Data reflects events logged in the first two weeks of the Reference Check A/B test on English Wikipedia. As with other Edit Check leading-indicator assessments, additional event volume will be required to confirm statistical significance. Full A/B test results will be reviewed in the forthcoming analysis for T400101.

11/21/25 Data Request Update:

• Initial PA Inquiry: Need editor data by 11/21/25 for design research.
• Data need discussed on Monday, 11/17/25: Generate two output lists (junior and experienced) for six wikis, ~1k records per wiki per list, using designated base data sources supplemented by previously used internal sources. Requirements included:
  • Exclude bots;
  • Follow legal, security, and privacy requirements;
  • Experienced editors defined as accounts that are at least 1 year old and active (at least 5 edits within the last month) and that have 500+ edits;
  • Junior editors defined as accounts with 10-110 edits.

On 11/18/25, after meeting with Daisy on Monday 11/17, I raised clarifying questions to confirm data availability, source validity.

Adding this quick update for tracking for future reference:

In T405177#11379529, @Sgs wrote:

In T405177#11369242, @Iflorez wrote:

Hola @Sgs:

• Constructive edit rate: the spec for this event says edit (made on mobile web within a user's first 24 hours [where the edit is not reverted within 48 hrs]), I am assuming this restrictions will be ensured in post analysis rather than by the instrumentation itself, because I'm seeing time criteria applied in the constructive edit rate metric query, for instance INTERVAL 48h, is this assumption correct?

Yes, the Constructive Edit rate metric will be calculated in the analysis/query. In analysis we can pull all edit_save events and determine whether an edit qualifies as a constructive edit, based on timing rules and revert status. To calculate Constructive Edit rate we need interaction data including edit_saved.

Is this also the case to determine if an edit was a revise-toneone or not? I cannot see any revise-tone tag in the metric catalog queries so I'm wondering if we need this information somewhere in the instrumentation or it will only be necessary on manual analysis instead of automated and that's why I cannot find it in the queries.

The metric/instrumentation planning was completed in the first week of November, and all related deliverables have been finalized.
Relevant materials are the Measurement plan, instrumentation specification, and the data instrumentation planning slides.
Onboarding instrumentation is currently listed for completion if time allows.

This work was completed in the first half of Q1. The dashboard is now available for on demand view.

Hola @Sgs:

• Constructive edit rate: the spec for this event says edit (made on mobile web within a user's first 24 hours [where the edit is not reverted within 48 hrs]), I am assuming this restrictions will be ensured in post analysis rather than by the instrumentation itself, because I'm seeing time criteria applied in the constructive edit rate metric query, for instance INTERVAL 48h, is this assumption correct?

The Indicator Plan of Action is now included in the Revise Tone Measurement Plan; See T397247.

This Measurement Plan work is now reviewed and complete.

See the interim dashboard here. I'll be sure to update this next week per the above.

a) Thank you @Samwilson for confirming the process had completed.

A "Post-merge build succeeded" update was posted on Oct 10 4:21 AM, that's the latest update on the patch.
The patch went out on the train yesterday.
I'm not yet seeing page-visited events when I query.

@TheresNoTime We need action_source and instrument_name info for both page_visited and click events.

Quick PA update:
I met with @OTichonova on 09/18 and and began developing the query for the supporting material. After drafting the query, I shared it with RDS teammates for review as well as the product subject area PM.
The final material was delivered on 09/22.

Inconclusive: At present we do not have the complete data picture and thus advise the team continue to monitor the backlog.

For this spike we looked at submission, deletion, and acceptance data as those could be anchored to aggregated data for accuracy comparison. Deletion data from the logging table and acceptance data from redirect, page, revision, comment, archive tables did not show a significant change or spike over time.

From the 2018 ACTRIAL work we learn the following tips:

Broad overview: The Articles for Creation (AfC) process is a quality control mechanism for new articles on English Wikipedia. It allows newer editors to submit article drafts for review by experienced editors before they are published. AfC draft submission = someone adding the special “AfC submission” template to their draft. Experienced editors use a [[ en.wikipedia.org/wiki/WP:AFCH | helper script ]] to evaluate drafts. The accurate way to know the first submission time is to a) collect data as it comes in or b) to look at the wikitext diffs of each draft’s history and find the exact edit where the AfC template first appeared, but the former is not possible for this task and the latter is time consuming and tricky post-hoc; instead of parsing all the diffs, we can use heuristics when looking backwards:

Quick PA update:
We received a two item batch request; this item was handled first.
I met with @dchen on 08/06 and the supportive material was provided the same day.
An updated request came in on 08/11 which was delivered on 08/15 after security engineering's approval and DM fine-tuning in Slack.
@dchen's feedback: "the final looks good, i will let you know if anything else comes up that could help refine any more."

Quick PA update:
We received a two item batch request; this was the second item.
I met with @dchen on 08/06 and and began developing the query for the supporting material once T395678 was completed.
After drafting the query, I shared it with RDS teammates for review.
An updated request came in on 08/11, and the final material was delivered on 08/15 after security engineering's approval and DM fine-tuning in Slack.
@dchen's feedback: "the final looks good, i will let you know if anything else comes up that could help refine any more."

7/28 Question on the Slack #growth-team channel:
data questions: are the two guidance systems instrumented? for instance:

• do we know if people (and how many) open the un-structured guidance once they are presented with the link eg. "More about this suggested edit"?
• do we know how many people skip (or not) the structured task guidance?
• are there any insights on conversation between the two guidance system? skipping/not skipping leading to an edit being published?
• is there any correlation between skipping/not skipping the guidance and reverts?

A) As of 01/24 there were three options for scheduling the data pipeline via Airflow:

1. Return to @mforns's POC for running Jupyter notebooks in Airflow (T322534#8467770), incorporating the code from @xcollazo's notebook to allow wmfdata-py to query MariaDB when running on worker nodes. T296661#9454074: As of 01/24 @mforns would like to see this option finished and utilized...if we could have that ready, that would be the best option; this project was deprioritized.
2. Abstract some of the nitty-gritty details in an easy-to-use function for querying against MariaDB replicas in an Airflow DAG and yank out all of the Python code out of the Welcome Survey aggregates notebook and into an Airflow DAG. (The "pure" approach.)
3. Ingest/sqoop the necessary data into the data lake and forget about running notebooks & querying replicas and just make it a pure PySpark-based job. Option 3 may @mpopov's preferred of three Airflow enabling paths.
4. Some of these might be enabled by an Airflow Jupyter Operator (status pending)

Also, getting an update/status on this comment will be pertinent: T296661#9454074

Quick updates:

• Data engineering notes—Heads-up: keep queries under 60 seconds as Superset will time out aggressively thereafter AND keep in mind that this will be an internal-access dashboard.
• @cmelo created a Superset dashboard in T399006 which has 7 charts (Superset virtual datasets) with last_x_months views.
• new-campaign-event-participants which is not reported on at a present and which is used for general team tracking, will be output by @cmelo on a monthly basis.

Update: Infrastructure unknowns with the echo notification job queue coupled with limited documentation and lack of echo notification maintainers means that the previously drafted 1.2.17 Get Started Experiment plan is now identified as a potential time drain (updated from a simple/quick change engineering endeavor). Further, it is misaligned with the new FY focus (~constructive edits for newer editors).
The team is now adapting the scope of the work (continuing with the 48hr time benchmark currently deployed) and is considering moving forward with this work as part of Experimentation Lab onboarding/transition, with an adapted approach (see New Approach) and paired down assessment component (CTR tracking).
Also, initially the team considered piloting this work on two wikis…then considered piloting on eswiki, arwiki, frwiki; now the team is considering a roll-out to all wikis.
I will stay posted to answer data decisions, and assist in planning where appropriate, as the team moves through this juncture.

This work is complete, here’s the report link.

This work is complete, here’s the report link.

Potential issues with running on Superset:

• Not all of the metrics can be calculated on Superset. The team will still need to grab the count of new-campaign-editors using another tool.
• Maintenance and data issues will likely apply at some point (see the highlighted chunk in the doc). Other issues may apply as well.

Background

A monthly first Friday run to sheets-output was the interim solution since T365404 was closed.

@KStoller-WMF I'll post a 1.2.9 blurb on the group Slack channel on Friday and will stay posted for questions.

@KStoller-WMF we ran out of time this morning at the demo for presenting this report. Can I share this report at the next demo (in two weeks) or would you prefer an async discussion in the Growth team channel?

Update: I presented the report findings with the Growth team this morning.

In T377098#10786715, @KStoller-WMF wrote:

Moving this to doing since @Iflorez is looking at data to determine when we will have enough experiment participants to calculate Constructive Activation.

@Iflorez Feel free to update or add a comment if that's not accurate.

Met with @MNeisler this afternoon to sync on the Tone Check area (this task, T397247, and T387918).
Key notes:

• WE 1.2 Tone Check AB Test Analysis (T387918) is currently under WE 1.1.2 and is likely to move to WE 1.1.
• WE 1.1 KR is now updated to: “WE1.1: Increase the rate at which editors with ≤100 cumulative edits publish constructive edits on mobile web by 4%, as measured by controlled experiments (by the end of Q2).”
• Reviewed the T387918 Measurement Plan in broad scope and discussed experiment details: important tags for analysis (revision tag: edit-check-shown, VE feature use: check-shown), existing group size estimation benchmark methods, key decision to be made via/with the experiment, analysis scope breakdowns.

This work was delivered and results were shared/discussed by moderator tools and trust and safety teams on two slack threads.

Team aim: Understand what people see in the UI - specifically admins and if they are seeing the No-JS version. We do not have browser session JS information at this time. However, we do know about local and global preferences related to JS settings on Recent Changes and Watchlist.

In the roll-up I'm capturing scenarios A and B as 'JS-disabled':
A:

Notes:

1. if rc/wl enhancedfilters disable is set globally, and local-exception is not checked, and the local setting is set to default, the up_value for the gp_property = 0
2. The second row in the table above is not possible to recreate. It is not possible to check local-exception without also setting the global option to disable.
3. global settings are queried in centralauth
4. storage priority goes to the latest non-default setting, that is what is reliably stored. That said, I did see that one of my return to default settings was updated in db: the updated property and the up_value as blank, containing neither 0 nor 1. Making a note of that here case it's helpful later.

The above query draft focuses on local settings. It does not take into account global settings and local exception overrides . @mpopov recommend's taking into account global preferences and override scenarios as well. This draft table aims to identify the scenarios and distinguish those to capture for counting as "JS disabled" accounts.

Active admins definition

The code that I'm working with today is below. I'm next going to confirm the active admins definition. From what I see from the movement metrics code, there is no threshold of activity counts but instead certain actions constitute the designation. I'm going to confirm this and then seek a code review in consultation hours.