Below is a proposed approach for for measuring how/if we've impacted constructive activation for mobile editors. This approach is based on discussions in the 8 October meeting as well as async input. Please see notes and this slack thread for more details on the approaches discussed.

To evaluate the impact we've caused in increasing constructive activation of mobile editors during the 2024-2025 fiscal year we will:

• Measure and report on the impact of each discrete intervention through a randomized controlled experiment. Each experiment will evaluate the impact of a deployed intervention on the constructive activation of mobile editors [1].
• Deploy interventions shown to have a positive impact on constructive activation to all wikis at a pace/sequence that volunteer alignment allows.
• Throughout the year, each team will maintain communication about when and where interventions are deployed so we can account for these deployments in other impact analyses if needed.
• At the end of the year, complete a year-over-year comparison to measure the collective impact of these widely deployed interventions.[2] In this analysis, we will measure June 2025 constructive activation rates and compare them to the June 2024 baseline rates as documented in T370726 to confirm if we achieved the targets identified for the KR (10% relative increase on mobile web, 25% relative increase on mobile apps).

1. Constructive activation rate is defined as the proportion of newcomers who publish ≥1 edit in the main namespace on a mobile device that is not reverted within 48 hours
2. As the collective impact analysis is based on a year-over-year comparison and not a controlled experiment, there may be other external factors that contribute to the observed changes that we will need to consider. We will use this year to help build a foundation we can learn from and iterate on for future collective experiments.

In T375926#10300625, @MNeisler wrote...

What you shared in T375926#10300625 looks great, Megan!

Question: when do you think we should plan to prioritize this measurement (T379285)?

Question: when do you think we should plan to prioritize this measurement (T379285)?

@ppelberg

It depends on what month we want to compare for the year over year analysis and timing of intervention deployments. See some timing considerations below:

• It should ideally be completed after all the interventions (with positive impact) have been scaled to all wikis.
• This analysis relies on mediawiki_history, which is provided at the beginning of each month. If we want to calculate the constructive activation rate for June 2025, we need to wait for the July mediawiki_history snapshot, because we require a full 24 hours of data into the next month (for users that registered at the end of June). This means the analysis in T379285 be prioritized for the beginning of August once the July snapshot is available.