Page MenuHomePhabricator

WE 1.2: Set percentage improvement target for constructive activation (FY 24/25)
Closed, ResolvedPublic

Description

In T360829, we established the baseline/existing constructive activation rate.

In this task, we will define how much the teams contributing to the Wiki Experiences Objective will strive to increase the constructive activation rate by.

Methodology

We will establish this improvement target by doing the following:

  1. Review the magnitude of impact past interventions have had on improving constructive activation. This work will happen in: T371737.
  2. Review how the constructive activation rate has varied over time. This work will happen in T370726.
  3. Reflect on what we learned through "1)" and "2)" to set an improvement target and assign a level of confidence to that improvement target [i]

i. Assigning a level of confidence to the improvement target we converge on will help us decide what kinds of interventions to prioritize. E.g. if we have a high confidence in our ability to affect constructive activation, we might prioritize larger "bets" that require more time before the impact of them can be known. Whereas if we have a lower confidence, we might prioritize smaller "bets" that return information more quickly so that we can develop a foundational understanding of the forces that impact constructive activation.

Event Timeline

@ppelberg

Below are some initial KR target proposals for mobile web and apps for review and discussion. These proposals are based on the insights from past interventions documented in T371737 and historical and seasonal trends documented in T370726.

Note: Per some past discussions, I'd recommend we propose separate KR targets for mobile web and apps given the significantly different baseline constructive activation rates and planned interventions.

Mobile Web: 15% relative increase (up to 20% as a more challenging target for consideration).

Rationale: This would account for an expected 8% to 9% YoY increase based on historical trends. It also aligns with past 13.5% and 17% relative increases in mobile web constructive activation rates that we've achieved through past small structured task interventions. We can potentially target higher depending on prioritized interventions but need to account for some potential uncertainty in the success of interventions based on newcomer context and how multiple edit checks and structured tasks may impact an editor's workflow.

Mobile Apps: 30% relative increase

Rationale: Mobile apps constructive activation rates have been fairly stable, hovering right around 2% for the past 4 years, but YoY trends can still fluctuate significantly with small changes due to the small observed rates. A 30% relative increase would equate to about a 0.6 percentage point increase (2.1% baseline to 2.7% after interventions). While this is a small absolute percentage point difference, I believe that it would be high enough to indicate an impact beyond typical YoY and seasonal fluctuations.

There is a little more uncertainty in the degree of impact we will be able to cause on mobile apps as I have not yet found a past intervention that sought to impact this metric specifically. If we feel more confident that the prioritized interventions may have a larger impact, we might consider raising this target to at least 40% (1 percentage point increase).

Update
@MNeisler and I are using the findings from T371726#10113802, T370726#10096610, and T371737#10100308 to propose an improvement target, the rationale for which we're drafting here: KR Proposal/WE 1.2 FY 24-25

Updates

Improvement targets:

  • Mobile web: 10% relative increase (y-o-y)
    • Note: naturally, we expect CA to increase by 6.5% y-o-y, per T370726
  • Mobile apps: 25% relative increase (y-o-y)