Page MenuHomePhabricator

Release Plan for Growth's notification A/B test
Closed, ResolvedPublic1 Estimated Story Points

Description

User Story:

As the Growth team, we want to ensure we are prepared to utilize xLab and that notification interactions are properly instrumented and understood, so that we can make more informed decisions about notifications and onboarding new editors.

As a new editor, I want to receive helpful guidance and encouragement when I get started on Wikipedia, so that I can become a successful contributor.

Details:

GrowthExperiments sends notifications to newcomers to reinforce them and improve engagement and re-engagement (as part of the feature Positive_reinforcement#Levelling_up ). In the past the impact of the notifications was analyzed in an experiment form, comparing the activation of users receiving the notification vs users not getting it.

As part of the SDS 2.1.6 hypothesis Growth wants to validate to different use cases of xLab, including an A/B test experiment to explore different messaging for the notifications.

Acceptance Criteria:

  • Setup the “Notifications experiment” via https://mpic.wikimedia.org/, start A/B test in testwiki
  • Allow time for QA testing in testwiki
  • Turn on feature flag in pilot wikis
  • Start A/B test on Growth pilot wikis on Tuesday 16th via https://mpic.wikimedia.org/
    • Pilot wikis: eswiki, arwiki, enwiki, frwiki

Event Timeline

KStoller-WMF moved this task from Inbox to Up Next (estimated tasks) on the Growth-Team board.
Restricted Application added a subscriber: alaa. · View Herald TranscriptSep 9 2025, 1:54 PM
KStoller-WMF set the point value for this task to 1.Sep 9 2025, 4:52 PM
KStoller-WMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.

Change #1190703 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[operations/mediawiki-config@master] Growth: enable new notifications

https://gerrit.wikimedia.org/r/1190703

Change #1190702 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[operations/mediawiki-config@master] Growth [testwiki]: enable new notifications and reduce scheduling time

https://gerrit.wikimedia.org/r/1190702

Change #1190702 merged by jenkins-bot:

[operations/mediawiki-config@master] Growth [testwiki]: enable new notifications and reduce scheduling time

https://gerrit.wikimedia.org/r/1190702

Mentioned in SAL (#wikimedia-operations) [2025-09-25T07:22:25Z] <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1190702|Growth [testwiki]: enable new notifications and reduce scheduling time (T404085)]], [[gerrit:1189617|mswikiquote: set timezone, sitename and project namespace (T404700)]], [[gerrit:rOPUP1189616a9f95|mswikiquote: add logo (T404700)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-25T07:28:47Z] <jforrester@deploy1003> anzx, jforrester, sgimeno: Backport for [[gerrit:1190702|Growth [testwiki]: enable new notifications and reduce scheduling time (T404085)]], [[gerrit:1189617|mswikiquote: set timezone, sitename and project namespace (T404700)]], [[gerrit:rOPUP1189616a9f95|mswikiquote: add logo (T404700)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-25T07:36:02Z] <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1190702|Growth [testwiki]: enable new notifications and reduce scheduling time (T404085)]], [[gerrit:1189617|mswikiquote: set timezone, sitename and project namespace (T404700)]], [[gerrit:rOPUP1189616a9f95|mswikiquote: add logo (T404700)]] (duration: 13m 36s)

Change #1190703 merged by jenkins-bot:

[operations/mediawiki-config@master] Growth: enable new notifications

https://gerrit.wikimedia.org/r/1190703

Mentioned in SAL (#wikimedia-operations) [2025-09-29T20:25:05Z] <sgimeno@deploy2002> Started scap sync-world: Backport for [[gerrit:1190703|Growth: enable new notifications (T404085)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-29T20:32:25Z] <sgimeno@deploy2002> sgimeno: Backport for [[gerrit:1190703|Growth: enable new notifications (T404085)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-29T20:38:50Z] <sgimeno@deploy2002> Finished scap sync-world: Backport for [[gerrit:1190703|Growth: enable new notifications (T404085)]] (duration: 13m 45s)

The experiment is live since September 29th 22:30 UTC+2. The Growth KPI dashboard shows appropriate bucketing for each group. We are already seeing computed CTRs in the automated analysis dashboard but that traffic is still from notifications scheduled prior to the deployment that are all getting the "control" messaging. Traffic should be analyzed from October 1st ~22:30 UTC+2.

This task’s QA is covered under T400118 (Personalized 48h notifications), which includes the variant experiments (T404085, T401308) and the core delivery flows.

That makes sense. I think we can consider this task resolved then. Thanks!