Page MenuHomePhabricator

Increase target audience for Surfacing Structured Task Experiment
Closed, ResolvedPublic2 Estimated Story Points

Description

User story & summary:

As a new editor, I need guidance and structure when completing my first edit on wiki, because then I'm more likely to constructively activate (complete my first edit that isn't reverted) and continue to edit.

Background & research:

The "Add a link" experiment shows this Structured task leads to increases in newcomer participation, particularly by making constructive (non-reverted) article edits:

  • an increase in the probability that newcomers make their first article edit (+16.6% over baseline, +10.1% over the unstructured add links task)
  • And increase in the probability that they are retained as newcomers (+16.2% over baseline, +5.7% over the unstructured link task)
Decision:

Currently, "Add a Link" suggestions are not available for high-traffic pages, limiting the number of newcomers exposed to these suggestions. To gather sufficient data for statistically significant results, we aim to expand the experiment audience.

  • Instead of restricting the experiment to logged-in accounts with 0 edits, we propose aligning it with the Edit Check audience, meaning it would include logged-in users with fewer than 100 edits.

Key result WE1.1: Increase constructive edits [i] by X% for editors with less than 100 cumulative edits, as measured by experiments by the end of Q2.

i. "Constructive edits" = edits that are not reverted within 48 hours of being published
Acceptance Criteria:
  • Add instrumentation for edit count
  • Engineering: Update experiment targeting to expand the audience from <1 edit to <100 edits.

Related work for Growth's PM:

  • Product: Discuss this proposal with Growth Ambassadors and Movement Communications
  • Product: Discuss the proposal with Product Analytics

Event Timeline

KStoller-WMF triaged this task as High priority.
KStoller-WMF moved this task from Inbox to Backlog on the Growth-Team board.
KStoller-WMF added subscribers: Trizek-WMF, Michael, Sgs.

Notes:

  • increasing targeting of audience included in experiment
  • edit count might need to be instrumented (are we already tracking this?)

The edit count is not added to the current instrumentation events, so if that's relevant, we should start tracking it before increasing the audience. I can file a task for it.

The edit count is not added to the current instrumentation events, so if that's relevant, we should start tracking it before increasing the audience. I can file a task for it.

Irene and I discussed, and she's considering this need.
@Iflorez - let us know if you have questions or want to meet to sync on the revised measurement plan?

KStoller-WMF set the point value for this task to 2.

Change #1128917 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] analytics(GrowthExperimentsInteractionLogger): add edit_count to the event data

https://gerrit.wikimedia.org/r/1128917

We're adding the edit_count value to the events tracked in mediawiki.product_metrics.growth_product_interaction, is that enough for the analysis edit count bucketing, or we should add the edit count also to the events logged in mediawiki.structured_task.article.link_suggestion_interaction, cc @Iflorez?

Change #1128920 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] feat(SurfacingStructuredTasks): increase max edit cap to 100

https://gerrit.wikimedia.org/r/1128920

Change #1128917 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] analytics(GrowthExperimentsInteractionLogger): add edit_count to the event data

https://gerrit.wikimedia.org/r/1128917

Is it ok to enable the audience increase anytime and capturing that time with a comment in this task. Or should we schedule this for a particular day/time, preferably coinciding with a backport window. cc @KStoller-WMF @Iflorez

@Sgs Can you share about benefits for going with the backport option?

Or should we schedule this for a particular day/time, preferably coinciding with a backport window.

We're adding the edit_count value to the events tracked in mediawiki.product_metrics.growth_product_interaction, is that enough for the analysis edit count bucketing, or we should add the edit count also to the events logged in mediawiki.structured_task.article.link_suggestion_interaction, cc @Iflorez?

Data gathered from each table:
mediawiki.product_metrics.growth_product_interaction - variant, user_id, dt, db
mediawiki.structured_task.article.link_suggestion_interaction - experiment indicator data: dt, impressions, viewport impressions, yes_clicks

For final analysis per the measurement plan, adding edit_count to the mediawiki.product_metrics.growth_product_interaction table will suffice. However, I think that your team will want to have an edit_count slice available on the indicator data for monitoring along the way and I recommend adding this data to both.

Change #1129819 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.21] analytics(GrowthExperimentsInteractionLogger): add edit_count to the event data

https://gerrit.wikimedia.org/r/1129819

Change #1129820 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.21] feat(SurfacingStructuredTasks): increase max edit cap to 100

https://gerrit.wikimedia.org/r/1129820

Change #1128920 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] feat(SurfacingStructuredTasks): increase max edit cap to 100

https://gerrit.wikimedia.org/r/1128920

Change #1129819 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.21] analytics(GrowthExperimentsInteractionLogger): add edit_count to the event data

https://gerrit.wikimedia.org/r/1129819

Change #1129820 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.21] feat(SurfacingStructuredTasks): increase max edit cap to 100

https://gerrit.wikimedia.org/r/1129820

Mentioned in SAL (#wikimedia-operations) [2025-03-20T13:28:14Z] <sgimeno@deploy2002> Started scap sync-world: Backport for [[gerrit:1129819|analytics(GrowthExperimentsInteractionLogger): add edit_count to the event data (T388622)]], [[gerrit:1129820|feat(SurfacingStructuredTasks): increase max edit cap to 100 (T388622)]]

Mentioned in SAL (#wikimedia-operations) [2025-03-20T13:31:06Z] <sgimeno@deploy2002> sgimeno: Backport for [[gerrit:1129819|analytics(GrowthExperimentsInteractionLogger): add edit_count to the event data (T388622)]], [[gerrit:1129820|feat(SurfacingStructuredTasks): increase max edit cap to 100 (T388622)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-03-20T13:39:14Z] <sgimeno@deploy2002> Finished scap sync-world: Backport for [[gerrit:1129819|analytics(GrowthExperimentsInteractionLogger): add edit_count to the event data (T388622)]], [[gerrit:1129820|feat(SurfacingStructuredTasks): increase max edit cap to 100 (T388622)]] (duration: 11m 00s)

The audience increase has been released today at 13:39 UTC along with the instrumentation update to capture edit_count in the mediawiki.product_metrics.growth_product_interaction events.

@Sgs Can you share about benefits for going with the backport option?

Or should we schedule this for a particular day/time, preferably coinciding with a backport window.

Backport windows are three slotted events during the day which allow us to release new features/changes. As opposed to train deployments which happen once per week. The main benefit is the Growth engineer has control over the feature/change release and the team can decide on which day/time, instead of waiting for the train deploy to happen and then monitor things.

For final analysis per the measurement plan, adding edit_count to the mediawiki.product_metrics.growth_product_interaction table will suffice. However, I think that your team will want to have an edit_count slice available on the indicator data for monitoring along the way and I recommend adding this data to both.

That makes sense, I will make a change for that asap.

Checked on testwiki wmf.21 - users with >1 edit can see Surfaced add link suggestions.

@KStoller-WMF - the engineering criteria and instrumentation criteria are done. Pinging you for review whether other items listed in the task description are done.

KStoller-WMF updated the task description. (Show Details)

Thanks! That was mostly a reminder to me to make sure I update our project page.
I've added that to my to-do list for this week, and we can consider this task done.