This task is about implementing the changes necessary to ensure people are being bucketed in the New Discussion Tool A/B test (T277825) as expected.
Requirements
Bucketing Criteria
People who meet all of the "Conditions" listed below, ought to have a 50% chance of being bucketed into the A/B test's control or test group. Bucketing should be done on a per Wikipedia basis.
- Conditions
- People who are editing at the Wikipedias listed in the ===Participating wikis section of T277825.
- People who are editing as logged in or logged out users
- People should remain the same group they were bucketed in for the duration of the test (even if they explicitly turn on or off the tool in their preferences)
- People who have NOT used the New Discussion Tool before, as determined by the state of the discussiontools-newtopictool-opened preference.
- People who have explicitly enabled or disabled the overall Discussion tools beta feature
Meta
- In order for us to identify any non-discussion tool EditAttemptStep events by users in the New Discussion Tool A/B test (T277825), the **Bucketing criteria** will need to be implemented such that the following is true:
- We are able to distinguish between A/B test and non-A/B test events within EditAttemptStep
- Within the New Discussion A/B test, we are able to distinguish between Discussion Tool and non-Discussion Tool events.
- Within the New Discussion Tool A/B test, we are able to distinguish all events logged for the control group and the test group
- This includes both Discussion Tools and VisualEditor/WikiEditor events
- The test bucket a person is assigned to should be reflected in/consistent with the state of the quick topic adding setting within Special:Preferences#mw-prefsection-editing-discussion
- Read: if you are in the test group, we assume the New Discussion Tool will be enabled for you by default. As such, the Quick topic adding in Special:Preferences#mw-prefsection-editing-discussion should show as being "enabled".
Note: we implemented something similar to above for the Reply Tool A/B test in T273096.
Minimal test case
- Log in to wiki as a user who has never used the new topic tool before (or set their DT beta preference)
- Visit a talk page (because bucketing happens the first time we actually need to check whether you qualify for the feature)
- In JS console enter mw.config.get( 'wgDiscussionToolsABTestBucket' )
- This should show "control", "test", or "". (The first two are in-test, the last means you've never been bucketed)
You may need to create a few accounts to verify that you do get assigned to different buckets.
Open questions
- 1. What – if any – additional work will need to be done to include logged out users within the A/B test?
- See T291307#7567176.
- 2. What – if any – client-side QA will need to be done the bucketing this ticket is asking for has been implemented as expected?