Page MenuHomePhabricator

Make config change to start Topic Subscriptions A/B Test
Closed, ResolvedPublic

Description

This task is about enabling/turning on the Topic Subscriptions A/B test at the ===Participating wikis (listed below).

Deployment timing

The A/B test will began on Thursday, 2 June 2022.

Participating wikis

Copied from T304027.

IDWikiCode
1.Amharic Wikipediaamwiki
2.Egyptian Wikipediaarzwiki
3.Bengali Wikipediabnwiki
4.Spanish Wikipediaeswiki
5.Persian Wikipediafawiki
6.French Wikipedia
7.Hebrew Wikipediahewiki
8.Hindi Wikipediahiwiki
9.Indonesia Wikipediaidwiki
10.Italian Wikipediaitwiki
11.Japanesejawiki
12.Korean Wikipediakowiki
13.Dutch Wikipedianlwiki
14.Oromo Wikipediaomwiki
15.Polish Wikipediaplwiki
16.Portuguese Wikipediaptwiki
17.Thai Wikipediathwiki
18.Ukrainian Wikipediaukwiki
19.Vietnamese Wikipediaviwiki
20.Chinese Wikipediazhwiki

Done

  • @ppelberg to document ===Deployment timing
  • @MNeisler Verify people at the ===Participating wikis are being bucketed in the ways we defined in T304030

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenMNeisler
ResolvedMNeisler
ResolvedMNeisler
ResolvedWhatamidoing-WMF
ResolvedDLynch
ResolvedDLynch
Resolved ppelberg
Resolvedmpopov
ResolvedMNeisler
ResolvedDLynch
ResolvedWhatamidoing-WMF
ResolvedRyasmeen
Resolved ppelberg
Resolved ppelberg
ResolvedDLynch
Resolved ppelberg
ResolvedDLynch
Resolved ppelberg
Resolved ppelberg
Resolvedmatmarex
ResolvedBUG REPORTmatmarex
Resolved ppelberg
ResolvedDLynch
ResolvedWhatamidoing-WMF
Resolved ppelberg
ResolvedSpikeMNeisler
Resolved ppelberg
ResolvedDLynch
ResolvedWhatamidoing-WMF
Resolved ppelberg
DeclinedMNeisler
ResolvedDLynch
Resolvedmatmarex

Event Timeline

Note: the A/B test starting is blocked on T304030 being verified.

Change 801818 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Launch DiscussionTools topic subscriptions a/b test

https://gerrit.wikimedia.org/r/801818

matmarex added a subscriber: matmarex.

(not scheduled for deployment yet, just prepating patches)

Participating wikis

This section will be considered final once T304027 is finalized.

Well, is it?

(not scheduled for deployment yet, just prepating patches)

Understood. Thank you for the heads up.

Participating wikis

This section will be considered final once T304027 is finalized.

Well, is it?

Yes. We can assume the 20 Wikipedias listed in T304027 will be the wikis participating in the A/B test. I've updated the task description to reflect this.

cc @Whatamidoing-WMF

Change 801818 merged by jenkins-bot:

[operations/mediawiki-config@master] Launch DiscussionTools topic subscriptions a/b test

https://gerrit.wikimedia.org/r/801818

Mentioned in SAL (#wikimedia-operations) [2022-06-02T13:25:39Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 3c12e779707e3982f973641e2b9c2522a429830f: Launch DiscussionTools topic subscriptions a/b test (T304029) (duration: 03m 16s)

MNeisler triaged this task as High priority.Fri, Jun 3, 2:32 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

I've checked the aggregate data and confirmed the bucketing appears to be working as expected based on the requirements defined in T304030, with one small exception noted below:

POTENTIAL ISSUE TO RESOLVE:

  • @DLynch When checking the data, I noticed that there were 7 edit attempts (5 attempts in the control group, 1 in the test) by a logged-out user (user_id = 0) that were bucketed in the AB test. This is a very small fraction of events logged and I can filter them out so I'm not too concerned about impacts on the AB test but wanted to flag here in case you might know the source. All of these were edit attempts made with VE and integration= 'page' in case that helps.

CONFIRMED CHECKS:
✅ Buckets are balanced. Since the deployment of the test on 2 June 2022 through today 8 June 2022, there is an expected close to 50/50 split of users between the control and test groups. 7168 users have made an edit attempt in the control group and 7208 users have made an edit attempt in the test group.
✅ Confirmed only people at the identified participating wikis are included in the test.
✅ Confirmed that people remain in the same group they were bucketed in for the duration of the test.
✅ Confirmed that both people who have and have not used the Topic subscriptions prior to the AB test are included and we can decipher these users with the current instrumentation
✅ Confirmed we can distinguish all events (including topics, comments and responses) for the control or test group.
✅ Confirmed events are logged in a way we can calculate KPI for both the control and test groups.

QA Notebook

I've checked the aggregate data and confirmed the bucketing appears to be working as expected based on the requirements defined in T304030, with one small exception noted below:

Excellent, @MNeisler. As you and I talked about during today's meeting, while the potential issue you name seems curious, we're going to NOT look into it further for two reasons:

  1. We don't anticipate this oddity to impact the test data and resulting analysis because we will be able to filter out these event
  2. It's a very small fraction of all logged edit attempts which leads us to not be concerned about it being symptomatic of a larger issue