Page MenuHomePhabricator
Create Task
Maniphest T304029

Make config change to start Topic Subscriptions A/B Test
Closed, ResolvedPublic
Actions

Description

This task is about enabling/turning on the Topic Subscriptions A/B test at the ===Participating wikis (listed below).

Deployment timing

The A/B test will began on Thursday, 2 June 2022.

Participating wikis

Copied from T304027.

IDWikiCode
1.Amharic Wikipediaamwiki
2.Egyptian Wikipediaarzwiki
3.Bengali Wikipediabnwiki
4.Spanish Wikipediaeswiki
5.Persian Wikipediafawiki
6.French Wikipedia
7.Hebrew Wikipediahewiki
8.Hindi Wikipediahiwiki
9.Indonesia Wikipediaidwiki
10.Italian Wikipediaitwiki
11.Japanesejawiki
12.Korean Wikipediakowiki
13.Dutch Wikipedianlwiki
14.Oromo Wikipediaomwiki
15.Polish Wikipediaplwiki
16.Portuguese Wikipediaptwiki
17.Thai Wikipediathwiki
18.Ukrainian Wikipediaukwiki
19.Vietnamese Wikipediaviwiki
20.Chinese Wikipediazhwiki

Done

  • @ppelberg to document ===Deployment timing
  • @MNeisler Verify people at the ===Participating wikis are being bucketed in the ways we defined in T304030

Details

Due Date
May 31 2022, 3:00 PM
SubjectRepoBranchLines +/-
Launch DiscussionTools topic subscriptions a/b testoperations/mediawiki-configmaster+21 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT276497 Scale DiscussionTools to all projects
 OpenNoneT251207 [Epic] Scale DiscussionTools to all Wikipedias
 ResolvedppelbergT284489 Deploy topic subscriptions (desktop) as opt-out feature at all projects
 ResolvedppelbergT314692 [Release Ticket] Offer Topic Subscriptions by default at A/B test wikis (desktop)
 ResolvedppelbergT280897 Run A/B test to evaluate impact of topic subscriptions
 ResolvedMNeislerT304029 Make config change to start Topic Subscriptions A/B Test
 ResolvedMNeislerT307733 Update PrefUpdate Allowlist to include topic subscription user preferences
 Resolved• Whatamidoing-WMFT304027 Identify wikis for Topic Subscriptions A/B test
 ResolvedDLynchT304030 Implement Topic Subscriptions A/B test bucketing
 ResolvedDLynchT302515 Unexpected DB insert from DiscussionTools::determineUserABTestBucket during GET request
 ResolvedppelbergT290517 [Release Ticket] Introduce New Topic Tool (desktop) as opt-out at A/B test wikis
 ResolvedmpopovT277825 Run A/B test to evaluate impact of New Topic Tool
 ResolvedMNeislerT290204 [SPIKE] Estimate New Discussion Tool A/B Test Feasibility
 ResolvedDLynchT291308 Make config change to start New Discussion Tool A/B Test
 Resolved• Whatamidoing-WMFT291306 Identify wikis for New Discussion Tool A/B test
 ResolvedRyasmeenT291307 Implement New Discussion Tool A/B test bucketing
 ResolvedppelbergT291536 [SPIKE] How can we ensure that we are able to identify and distinguish prior users of the New Discussion Tool in the AB test?
 ResolvedppelbergT292120 [SPIKE] Determine how we can identify blocked save attempts
 ResolvedDLynchT293962 Introduce a "hint" that informs people how they can enable the legacy section=new experience
 ResolvedppelbergT294491 [SPIKE] How many people have turned off the New Discussion Tool since it's been available as a Beta Feature
 ResolvedDLynchT294167 Implement way to identify people who have opened the New Discussion Tool before
 ResolvedppelbergT297847 Ensure New Discussion Tool Hint Is Fully Translated
 ResolvedppelbergT301496 Add distinct event to talk_page_edit for when a new section is added
 ResolvedmatmarexT288741 Talk Page Empty State: write English copy for translation
 ResolvedBUG REPORTmatmarexT291085 redlink=1 not respected in zhwiki
 ResolvedppelbergT291535 Inform participating wikis of start of New Discussion Tool A/B Test
 ResolvedDLynchT291873 Deploy config change to "turn off" New Discussion Tool A/B test
 Resolved• Whatamidoing-WMFT295098 [SPIKE] What – if any – reactions have people had to the start of the New Discussion Tool A/B test?
 ResolvedppelbergT297714 [SPIKE] What is the makeup of people who have adjusted the state of the Discussion tools beta feature?
 ResolvedSpikeMNeislerT301495 [SPIKE] Investigate what init type is associated with the sessions where a ready event is being emitted, but an anonymous user token is not being logged
 ResolvedppelbergT301497 Ensure the A/B test bucket is logged when a logged out user edits a full page in wikitext
 ResolvedDLynchT301499 [SPIKE] What could be causing a ready event to be emitted, but an anonymous user token NOT to be logged
 Resolved• Whatamidoing-WMFT304658 Announce New Topic Tool A/B test findings to wikis participating in test
 ResolvedppelbergT306481 Add more granular experience level buckets to New Topic Tool A/B test report
 DeclinedMNeislerT306579 Investigate rate at which Senior Contributors abandon New Topic Tool edits
ResolvedDLynchT307410 [Config Change] Enable the New Topic Tool as opt-out at A/B test wikis
 ResolvedmatmarexT295425 [Config Change] Deploy Automatic Topic Subscriptions as Beta Feature at Remaining Wikis

Event Timeline

ppelberg created this task.Mar 16 2022, 11:39 PM
ppelberg mentioned this in T304030: Implement Topic Subscriptions A/B test bucketing.
ppelberg updated the task description. (Show Details)Mar 16 2022, 11:43 PM
ppelberg updated the task description. (Show Details)
ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.
ppelberg moved this task from Incoming to Upcoming on the Editing-team (Kanban Board) board.
ppelberg mentioned this in T304031: Deploy config change to "turn off" Topic Subscriptions A/B test.
ppelberg added a subscriber: MNeisler.
ppelberg mentioned this in T280897: Run A/B test to evaluate impact of topic subscriptions.Apr 13 2022, 8:26 PM
ppelberg added a subtask: T307733: Update PrefUpdate Allowlist to include topic subscription user preferences.May 6 2022, 11:27 PM
ppelberg moved this task from Upcoming to Ready to Be Worked On on the Editing-team (Kanban Board) board.May 17 2022, 12:45 AM
ppelberg updated the task description. (Show Details)May 17 2022, 1:45 AM
ppelberg updated the task description. (Show Details)
ppelberg added a subtask: T304027: Identify wikis for Topic Subscriptions A/B test.
ppelberg added a subtask: T302515: Unexpected DB insert from DiscussionTools::determineUserABTestBucket during GET request.
ppelberg removed a subtask: T302515: Unexpected DB insert from DiscussionTools::determineUserABTestBucket during GET request.
ppelberg added a subtask: T304030: Implement Topic Subscriptions A/B test bucketing.
ppelberg added a subtask: T290517: [Release Ticket] Introduce New Topic Tool (desktop) as opt-out at A/B test wikis.
MNeisler closed subtask T307733: Update PrefUpdate Allowlist to include topic subscription user preferences as Resolved.May 17 2022, 2:31 PM
ppelberg set Due Date to May 31 2022, 3:00 PM.May 17 2022, 9:54 PM
ppelberg closed subtask T290517: [Release Ticket] Introduce New Topic Tool (desktop) as opt-out at A/B test wikis as Resolved.May 27 2022, 1:04 AM
ppelberg added a comment.May 27 2022, 10:19 PM

Note: the A/B test starting is blocked on T304030 being verified.

ppelberg added a project: Goal.May 27 2022, 10:19 PM
ppelberg mentioned this in T295425: [Config Change] Deploy Automatic Topic Subscriptions as Beta Feature at Remaining Wikis.May 27 2022, 10:39 PM
gerritbot added a comment.May 31 2022, 9:45 PM

Change 801818 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Launch DiscussionTools topic subscriptions a/b test

https://gerrit.wikimedia.org/r/801818

gerritbot added a project: Patch-For-Review.May 31 2022, 9:45 PM
matmarex moved this task from Ready to Be Worked On to Doing on the Editing-team (Kanban Board) board.May 31 2022, 9:45 PM
matmarex subscribed.

(not scheduled for deployment yet, just prepating patches)

Participating wikis

This section will be considered final once T304027 is finalized.

Well, is it?

ppelberg added a subscriber: • Whatamidoing-WMF.May 31 2022, 9:56 PM
In T304029#7972345, @matmarex wrote:

(not scheduled for deployment yet, just prepating patches)

Understood. Thank you for the heads up.

Participating wikis

This section will be considered final once T304027 is finalized.

Well, is it?

Yes. We can assume the 20 Wikipedias listed in T304027 will be the wikis participating in the A/B test. I've updated the task description to reflect this.

cc @Whatamidoing-WMF

ppelberg updated the task description. (Show Details)May 31 2022, 9:58 PM
Restricted Application added subscribers: Ericliu1912, Huji, Stang and 2 others. · View Herald TranscriptMay 31 2022, 9:58 PM
ppelberg updated the task description. (Show Details)May 31 2022, 9:58 PM
Shizhao added a project: Chinese-Sites.Jun 1 2022, 3:45 AM
Shizhao moved this task from Backlog to Extensions/Skins on the Chinese-Sites board.Jun 1 2022, 3:47 AM
ppelberg added a subtask: T295425: [Config Change] Deploy Automatic Topic Subscriptions as Beta Feature at Remaining Wikis.Jun 1 2022, 11:02 PM
matmarex added a comment.Jun 1 2022, 11:18 PM

Scheduled for tomorrow: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20220602T1300

matmarex claimed this task.Jun 1 2022, 11:18 PM
ppelberg awarded a token.Jun 2 2022, 1:02 AM
gerritbot added a comment.Jun 2 2022, 1:12 PM

Change 801818 merged by jenkins-bot:

[operations/mediawiki-config@master] Launch DiscussionTools topic subscriptions a/b test

https://gerrit.wikimedia.org/r/801818

Stashbot added a comment.Jun 2 2022, 1:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-02T13:25:39Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 3c12e779707e3982f973641e2b9c2522a429830f: Launch DiscussionTools topic subscriptions a/b test (T304029) (duration: 03m 16s)

Maintenance_bot removed a project: Patch-For-Review.Jun 2 2022, 1:30 PM
matmarex moved this task from Doing to QA on the Editing-team (Kanban Board) board.Jun 2 2022, 1:56 PM
matmarex added a project: Editing QA.
ppelberg reassigned this task from matmarex to MNeisler.Jun 2 2022, 4:20 PM
ppelberg edited projects, added Editing-team (Tracking), Product-Analytics; removed Editing QA, Editing-team (Kanban Board).
ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.
MNeisler triaged this task as High priority.Jun 3 2022, 2:32 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.
ppelberg closed subtask T295425: [Config Change] Deploy Automatic Topic Subscriptions as Beta Feature at Remaining Wikis as Resolved.Jun 3 2022, 11:34 PM
ppelberg closed subtask T304030: Implement Topic Subscriptions A/B test bucketing as Resolved.Jun 6 2022, 5:08 PM
MNeisler added a subscriber: DLynch.Jun 8 2022, 2:58 PM

I've checked the aggregate data and confirmed the bucketing appears to be working as expected based on the requirements defined in T304030, with one small exception noted below:

POTENTIAL ISSUE TO RESOLVE:

  • @DLynch When checking the data, I noticed that there were 7 edit attempts (5 attempts in the control group, 1 in the test) by a logged-out user (user_id = 0) that were bucketed in the AB test. This is a very small fraction of events logged and I can filter them out so I'm not too concerned about impacts on the AB test but wanted to flag here in case you might know the source. All of these were edit attempts made with VE and integration= 'page' in case that helps.

CONFIRMED CHECKS:
✅ Buckets are balanced. Since the deployment of the test on 2 June 2022 through today 8 June 2022, there is an expected close to 50/50 split of users between the control and test groups. 7168 users have made an edit attempt in the control group and 7208 users have made an edit attempt in the test group.
✅ Confirmed only people at the identified participating wikis are included in the test.
✅ Confirmed that people remain in the same group they were bucketed in for the duration of the test.
✅ Confirmed that both people who have and have not used the Topic subscriptions prior to the AB test are included and we can decipher these users with the current instrumentation
✅ Confirmed we can distinguish all events (including topics, comments and responses) for the control or test group.
✅ Confirmed events are logged in a way we can calculate KPI for both the control and test groups.

QA Notebook

ppelberg closed this task as Resolved.Jun 8 2022, 8:23 PM
In T304029#7989257, @MNeisler wrote:

I've checked the aggregate data and confirmed the bucketing appears to be working as expected based on the requirements defined in T304030, with one small exception noted below:

Excellent, @MNeisler. As you and I talked about during today's meeting, while the potential issue you name seems curious, we're going to NOT look into it further for two reasons:

  1. We don't anticipate this oddity to impact the test data and resulting analysis because we will be able to filter out these event
  2. It's a very small fraction of all logged edit attempts which leads us to not be concerned about it being symptomatic of a larger issue
ppelberg updated the task description. (Show Details)Jun 8 2022, 8:26 PM
Shizhao moved this task from Extensions/Skins to Closed on the Chinese-Sites board.Jun 9 2022, 3:29 AM
Stang unsubscribed.Jun 9 2022, 5:25 AM
• Whatamidoing-WMF closed subtask T304027: Identify wikis for Topic Subscriptions A/B test as Resolved.Jun 16 2022, 7:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits