@GGalofre-WMF I've completed a one-off analysis updating and extending the experiment impact analysis completed in T429589. This provides some additional insights to supplement the overall experiment metrics available in the Test Kitchen automated analytics dashboard.

@GGalofre-WMF
To refine the funnel analysis and edit abandonment rate calculation, I’d like to clarify which steps intentionally block the user from creating the article.

Based on a review of existing events, we do not currently have sufficient existing instrumentation to answer the questions identified above. Events will need to be added to VisualEditorFeatureUse to track user engagement with the pre-populated article structure. In the meantime, we may need to rely on qualitative insights and potentially manual review of a sample of articles created by users in the treatment group.

@OBenhmida Yes, your summary and takeways are correct.

Hi @OBenhmida,
I completed an initial analysis estimating the volume of Event Pathways dialog triggers we can expect based on the editing activity across event worklist articles.

So for Article Guidance, based on the previous criteria the abandonment could be considered as: Sessions that enter the workflow - sessions that are blocked due to notability restrictions - sessions that publish the article.

Here's the chart showing weekly engagement count trends over the past year (limited to editors fewer than 1,000 lifetime edits and fewer than 25 edits in the past year).

Resolving this task as we've finalized a baseline and target for this KR. Please reopen if any additional questions come up.

Updated DE 1.2 Baseline and Target for weekly engagement count (limited to editors fewer than 1,000 lifetime edits and fewer than 25 edits in the past year).

I completed an additional exploratory analysis to identify the optimal filter thresholds needed to isolate users who have not yet formed a consistent editing habit, while minimizing baseline variance caused by outlier power users.

adjusting the article abandonment rate metric definition to use t_experiment.action = 'init' is a relatively quick fix. I will draft an MR to update this definition; once merged, the dashboard will reflect this updated metric definition."

Per discussions with @SonjaPerry and @Mayakp.wiki today, we'd like to limit the experiment-based metric to users who have not yet formed an editing habit, as that's the primary target group of this KR.

@GGalofre-WMF I've added the following three metrics to the dashboard to help track the impact on junior editors on mobile.

@MNeisler , as discussed today, could you pls do a check on the numbers if we exclude the outliers ie editors with <=1k edits per week, and <=10k edits per week.

For #3 "Filtering by junior/experienced and mobile/desktop" > define a metric titled "Article Guidance article survival rate (30 days) for Junior Editors on Mobile" I have added 3 metrics based on this to the metrics document highlighted in green here what would be the process to add them to the TKAA?

Thanks for clarifying the key additions needed here. I've proposed some initial suggested steps below based on what makes the most sense in the given timeframe:

Resolving as we've finalized the baseline and target for this KR.

@GGalofre-WMF I completed a more detailed funnel analysis to investigate specific user drop off points for users who open the CX dashboard to start or continue a translation. Let me know if you have any questions or any further breakdowns here would be useful.

Resolving this task as we have completed the initial modeling projections based on different pathways to becoming a contributor, including projected attrition rates and incorporation of readers' estimates.

I was curious why we are currently seeing such a significant decrease in Article Abandonment Rate (-59%), especially given the decrease of distinct users publishing an article in the treatment group (as indicated in T429589#12063429).

I completed a one-off analysis of the experiment data to provide some of the requested data points and more insights into overall trends. See summary below and please let me know if you have any questions:

@ppelberg Please see a summary of the leading indicator results below.
One key takeway: Edit suggestions have a clear, positive impact on edit completion rate and edit quality for users who view them. However, because the exposure rate is so low, we are not able to see these impacts in the overall experiment metrics.

@GGalofre-WMF:

The assumption I'm trying to confirm or refute is: When the tools work well, contributors publish and come back.
As a proxy for "encountered a blocker," I was thinking of sessions that included a template failure, an MT threshold rejection, or an abandoned translation with no return.

Below are some recommended targets for the DE 1.1 KR. These are based on a review of trends in the metric from January 2025 through April 2026, current baselines (detailed in T422743#12021964) and a review of past inteventions impact on similar type metrics.

Per discussions with @ppelberg, we've decided to include English Wikipedia in the calculated baseline.

Below are some recommended targets for the DE 1.4 KR. These are based on a review of trends in 30-day Article Survival Rate from February 2024 through April 2026, current baselines (detailed in T429018#12019992) and current experiment intervention trends.

Below are some recommended targets for the DE 1.3 KR. These are based on a review of trends in second week edit retention from May 2024 through May 2026, current baselines (detailed in T419460#12014666) and a review of past inteventions impact on retention rates.

@ifried - See below and let me know what target you'd like to proceed with or if you have any questions/additonal data points that would be helpful for deciding how to proceed.

I've updated the queries to use the new wmf_product.personal_dashboard_event_v1 dataset and applied the timestamp column optimizations suggested in T429051#12015515.

• Baseline (Excluding English Wikipedia): 0.64%
• Baseline (Including English Wikipedia): 0.40%

Some clarifications regarding the proposed definition and measurement approaches:

KR Definition

Proportion of registered users with 100 or fewer edits who publish as least one constructive edit on mobile web (edit was not reverted within 48 hours) on a Wikipedia main namespace within the experiment (or defined timeframe).

Please recalculate the baseline 30-day article survival rate for junior editors
Include: volume of articles, minimum sample size to detect a 5pp change and a 10 pp change, experiment duration estimates.

Thanks @KStoller-WMF and @Samwalton9-WMF.

@Mayakp.wiki @ifried

Definition/Categories

We aligned on Weekly Engagement Count as the name for the KR metric. Definition: Average actions[1] per user to any namespace in the past calendar week, whether reverted or not.

Thanks for checking in! I’ve had to shift focus to finalizing the FY26-27 metrics, but I plan to pivot back to this next week. Per your prioritized list (which I've copied to the task description), I'll focus first on providing details on the specific drop-off points and tackle the remaining data points as time allows. Let me know if any priorities have shifted in the meantime.

That said, if exposure to the interventions is already limited to specific audiences within the experiments, it may be reasonable for the metric itself to remain broader. I'm curious to hear your perspective and whether you'd recommend narrowing the KR measurement or keeping it as currently defined.

Given that Spanish is delayed and not part of the initial launch group, could you help us with a revised estimated experiment duration to reach statistical power with the remaining wikis.

I reviewed the second-week retention rates over the past year to identify the current baseline.

I completed an initial analysis of weekly edit rates to help align on how we measure this and begin identifying a baseline. The data covers Wikipedia edits made by registered users across all namespaces between April 2025 and April 2026. Let me know if this brings up any questions or concerns.

Hi @GGalofre-WMF See responses below:

Another thing I'm curious about is that the rate decreases over time

Just to make sure I understand - in the Average graph is each data point effectively "From the start date up to this date, what is the average engagement rate"?

I've updated the Superset dashboard with charts to track this KR metric. These are currently located on a separate tab ("Two-day engagement with diffs [DRAFT]") to help avoid timeout issues.