Implement bucketing for Usability Improvements A/B test
Closed, ResolvedPublic
Actions

Description

This task is about implementing the changes necessary to ensure people are being bucketed in the Usability Improvements A/B test (T302358) as expected.

Bucketing requirements

Bucketing should include ~~both logged-in and logged-out users~~ logged-in users
Everyone who is accessing a desktop talk page on any of the wikis listed in the Participating wikis, logged in ~~and logged out~~, should have a 50% chance of being included/bucketed into the A/B test's control or test group
Bucketing should be done on a per-Wikipedia basis. 50% of people within a given wiki should be placed within the control group; 50% should be bucketed in the test group
People should remain in the same group for the duration of the test (even if they explicitly turn on or off discussion tool preferences).
Testing should be done across various desktop skins (not limited to just Vector 2022)

Instrumentation-Related Requirements

Within the A/B test, we are able to distinguish all events logged for the control group and the test group.
For all wikis in the test, we are able to join events from DesktopWebUIActionsTracking to EditAttemptStep (confirmed in T330938)
We are able to track if/when someone in the test group adjusts any of the discussion tool-related preferences.
We can track the desktop skin type being viewed by a user in the AB test (confirmed in T335159)

Participating wikis
Defined in T332946 (still being finalized)

Details

	Subject	Repo	Branch	Lines +/-
	Update a/b test code for visual enhancements a/b test	mediawiki/extensions/DiscussionTools	wmf/1.41.0-wmf.7	+10 -24
	Update a/b test code for visual enhancements a/b test	mediawiki/extensions/DiscussionTools	master	+10 -24

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T276497 Scale DiscussionTools to all projects
Open	None	T366095 Deploy comment thanking to all wikis
Open	None	T310959 Scale Usability Improvements to all projects
Open	Trizek-WMF	T319146 [Release Ticket] Offer Usability Improvements as default-on feature at all projects (desktop)
Resolved	ppelberg	T302358 [A/B Test] Run an A/B test to evaluate impact of Usability Improvements
Resolved	ppelberg	T333715 Implement bucketing for Usability Improvements A/B test

Event Timeline

ppelberg created this task.Mar 31 2023, 5:58 PM

ppelberg edited projects, added Product-Analytics, Editing-team (Tracking); removed Editing-team.

VPuffetMichel moved this task from Backlog to Triaged on the DiscussionTools board.Apr 6 2023, 1:54 PM

MNeisler triaged this task as Medium priority.Apr 10 2023, 2:57 PM

MNeisler moved this task from Triage to Current Quarter on the Product-Analytics board.

MNeisler renamed this task from Implement bucketing requirements for Usability Improvements A/B test to Implement bucketing for Usability Improvements A/B test.Apr 26 2023, 6:45 PM

MNeisler updated the task description. (Show Details)

I'm not sure we can apply this to logged out users in any reliable way -- it's going to hit the same caching issue as we encountered with the last A/B test, in that it's a change to the page-content that'd be based on a cookie and so it'd just trigger at a page level and flip around every time the page fell out of the cache based on whatever user happened to view it first.

DLynch updated the task description. (Show Details)May 4 2023, 4:46 PM

Change 916902 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Update a/b test code for visual enhancements a/b test

https://gerrit.wikimedia.org/r/916902

gerritbot added a project: Patch-For-Review.May 8 2023, 6:40 AM

Change 916902 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Update a/b test code for visual enhancements a/b test

https://gerrit.wikimedia.org/r/916902

Maintenance_bot removed a project: Patch-For-Review.May 8 2023, 2:12 PM

Change 917160 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@wmf/1.41.0-wmf.7] Update a/b test code for visual enhancements a/b test

https://gerrit.wikimedia.org/r/917160

gerritbot added a project: Patch-For-Review.May 8 2023, 2:51 PM

MNeisler updated the task description. (Show Details)May 8 2023, 4:00 PM

Change 917160 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.41.0-wmf.7] Update a/b test code for visual enhancements a/b test

https://gerrit.wikimedia.org/r/917160

Mentioned in SAL (#wikimedia-operations) [2023-05-08T20:08:19Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:917160|Update a/b test code for visual enhancements a/b test (T333715)]], [[gerrit:916903|Enable DiscussionTools visual enhancements a/b test (T302358)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-08T20:09:49Z] <taavi@deploy1002> kemayo and taavi: Backport for [[gerrit:917160|Update a/b test code for visual enhancements a/b test (T333715)]], [[gerrit:916903|Enable DiscussionTools visual enhancements a/b test (T302358)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet

Maintenance_bot removed a project: Patch-For-Review.May 8 2023, 8:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-08T20:20:13Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:917160|Update a/b test code for visual enhancements a/b test (T333715)]], [[gerrit:916903|Enable DiscussionTools visual enhancements a/b test (T302358)]] (duration: 11m 54s)

MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.May 12 2023, 7:02 PM

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

I've QA'd the AB test events logged to date and confirmed that the numbers appear as expected based on a 50/50 split.

We start logging AB test events on 8 May 2023 for all participating wikis. Here are the number of AB test editing sessions and users that made an edit attempt since the deployment of the test on 8 May 2023 through 16 May 2023.