Page MenuHomePhabricator

Implement New Discussion Tool A/B test bucketing
Closed, ResolvedPublic

Description

This task is about implementing the changes necessary to ensure people are being bucketed in the New Discussion Tool A/B test (T277825) as expected.

Requirements

Bucketing Criteria
People who meet all of the "Conditions" listed below, ought to have a 50% chance of being bucketed into the A/B test's control or test group. Bucketing should be done on a per Wikipedia basis.

  • Conditions
    • People who are editing at the Wikipedias listed in the ===Participating wikis section of T277825.
    • People who are editing as logged in or logged out users
    • People should remain the same group they were bucketed in for the duration of the test (even if they explicitly turn on or off the tool in their preferences)
    • People who have NOT used the New Discussion Tool before, as determined by the state of the discussiontools-newtopictool-opened preference.
    • People who have explicitly enabled or disabled the overall Discussion tools beta feature

Meta

  • In order for us to identify any non-discussion tool EditAttemptStep events by users in the New Discussion Tool A/B test (T277825), the **Bucketing criteria** will need to be implemented such that the following is true:
    • We are able to distinguish between A/B test and non-A/B test events within EditAttemptStep
    • Within the New Discussion A/B test, we are able to distinguish between Discussion Tool and non-Discussion Tool events.
    • Within the New Discussion Tool A/B test, we are able to distinguish all events logged for the control group and the test group
      • This includes both Discussion Tools and VisualEditor/WikiEditor events
  • The test bucket a person is assigned to should be reflected in/consistent with the state of the quick topic adding setting within Special:Preferences#mw-prefsection-editing-discussion
    • Read: if you are in the test group, we assume the New Discussion Tool will be enabled for you by default. As such, the Quick topic adding in Special:Preferences#mw-prefsection-editing-discussion should show as being "enabled".

Note: we implemented something similar to above for the Reply Tool A/B test in T273096.

Minimal test case

  1. Log in to wiki as a user who has never used the new topic tool before (or set their DT beta preference)
  2. Visit a talk page (because bucketing happens the first time we actually need to check whether you qualify for the feature)
  3. In JS console enter mw.config.get( 'wgDiscussionToolsABTestBucket' )
  4. This should show "control", "test", or "". (The first two are in-test, the last means you've never been bucketed)

You may need to create a few accounts to verify that you do get assigned to different buckets.

Open questions

  • 1. What – if any – additional work will need to be done to include logged out users within the A/B test?
  • 2. What – if any – client-side QA will need to be done the bucketing this ticket is asking for has been implemented as expected?
    • @Ryasmeen will verify people are being bucketed within the client; a check of how people are being bucketed in aggregate will happen in T291308.

Done

  • All ===Open questions are answered
  • A patch has been deployed that meets what's described in the ===Requirements section above
  • @Ryasmeen has verified people are being bucketed (a check of how people are being bucketed in aggregate will happen in T291308)

Related Objects

Event Timeline

ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.
ppelberg added a subscriber: MNeisler.

In this context, "not used" is being defined as people whose discussiontools-editmode preference is empty.

This is also going to rule out people who have used the reply tool. Is that okay?

In this context, "not used" is being defined as people whose discussiontools-editmode preference is empty.

This is also going to rule out people who have used the reply tool. Is that okay?

Great spot. We'll answer this question in the newly-created T291536.

ppelberg updated the task description. (Show Details)

During yesterday's team meeting, @matmarex spotted the need for the state of the Quick topic adding setting in Special:Preferences to be consistent with how – if at all – the New Discussion Tool is configured for the test group you are in.

In line with the above, I've added the following to the task description's ===Requirements section:

  • The test bucket a person is assigned to should be reflected in/consistent with the state of the quick topic adding setting within Special:Preferences#mw-prefsection-editing-discussion
    • Read: if you are in the test group, we assume the New Discussion Tool will be enabled for you by default. As such, the Quick topic adding in Special:Preferences#mw-prefsection-editing-discussion should show as being "enabled".

Change 745672 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Update abtest bucketing for newtopictool usage

https://gerrit.wikimedia.org/r/745672

Next step

Change 745672 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Update abtest bucketing for newtopictool usage

https://gerrit.wikimedia.org/r/745672

ppelberg updated the task description. (Show Details)

Change 746931 had a related patch set uploaded (by DLynch; author: DLynch):

[operations/mediawiki-config@master] Enable A/B test for discussiontools new topic tool on beta cluster

https://gerrit.wikimedia.org/r/746931

@DLynch @ppelberg - Just to confirm: Are we planning to include both logged-out users and logged-in users in this AB test or just logged-in?

If including logged-out, would it be possible for us to use the existing anonymous_user_token field created in T221195#5260013 so we have a persistent identifier for anonymous contributors?

This is only for logged in users (because we're storing the ab test bucket and whether the user has used the new topic tool before as a user preference, which can't apply to logged out users).

Change 746931 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable A/B test for discussiontools new topic tool on beta cluster

https://gerrit.wikimedia.org/r/746931

This comment was removed by ppelberg.

Updates
Below are the outcomes of the discussion @DLynch, @MNeisler, and I had today about the A/B test inclusion criteria.

  1. DECIDED: People who have manually adjusted the state of the Discussion tools beta feature within Special:Preferences#mw-prefsection-betafeatures should NOT be included in the A/B test.
    • Thinking: we assume the majority of people who will have adjusted this preference will be Senior Contributors whose usage behavior we are less concerned with in this context. Reason: the primary objective of this A/B test is to learn whether the New Discussion Tool causes Junior Contributors to be more or less successful starting new discussions. More in T277825.
  2. DECIDED: Logged out users should be INCLUDED in the A/B test.

Next Steps

  • @DLynch to implement "1." and and then "2." above.
  • @ppelberg to decide whether the A/B needs to begin for logged in and logged out users at the same time
  • @ppelberg to decide whether it's worth @MNeisler expelling effort to check the makeup of the people who have explicitly adjusted the state of the Discussion tools beta feature within Special:Preferences#mw-prefsection-betafeatures

Update:

  1. Users are getting bucketed to "control" or "test".
  2. Accounts which used the new topic tool before, are not getting assigned to any buckets.

However I noticed a scenario where, if after creating a new account and logging in for the first time, the new user lands on a talk page (which was the last page used by the previous login session), they are not getting bucketed, even though it technically means visiting a talk page already. Refreshing the same page does assign a bucket to that user though. @DLynch is looking into it.

Change 748157 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Allow logged out users to be enrolled in the A/B test

https://gerrit.wikimedia.org/r/748157

Change 748169 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Ignore the beta feature setting when enrolling in the A/B test

https://gerrit.wikimedia.org/r/748169

However I noticed a scenario where, if after creating a new account and logging in for the first time...

T298287 now exists for the issue @Ryasmeen identified above.

Change 748157 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Allow logged out users to be enrolled in the A/B test

https://gerrit.wikimedia.org/r/748157

Change 748169 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Ignore the beta feature setting when enrolling in the A/B test

https://gerrit.wikimedia.org/r/748169

Re-tested this. After the last two patches:

  1. If a logged-out user visits a talk page, they are getting bucketed even if they are not logged in and
  2. Regardless of whether a user has the Beta feature for "New Discussion Tool" /"Quick Topic Adding" turned on/off, they are now getting bucketed when they visit a talk page in all cases.
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)

Re-tested this. After the last two patches:

  1. If a logged-out user visits a talk page, they are getting bucketed even if they are not logged in and
  2. Regardless of whether a user has the Beta feature for "New Discussion Tool" /"Quick Topic Adding" turned on/off, they are now getting bucketed when they visit a talk page in all cases.

Excellent, @Ryasmeen.

A follow up question for you: Would it be accurate for us to think people who HAVE used the New Discussion Tool before, as determined by the state of the mw.user.options.get( 'discussiontools-newtopictool-opened' ) preference [i], are being assigned to a test bucket?

...I ask the above because I am just now realizing that the the task description incorrectly stated [ii][iii] "People who have NOT used the New Discussion Tool before, as determined by the state of the discussiontools-newtopictool-opened preference." and I want to be doubly sure this requirement did NOT get implemented.


i. More context in T294167#7504331
ii. In T294167 we decided to include people in the A/B test regardless of whether they have or have not used the New Discussion Tool before
iii. I've now removed this requirement: T291307#7630896

@Ryasmeen: please disregard the question I posed in T291307#7630898 for now.

Before we do any further testing of the bucketing criteria, @DLynch, @MNeisler, and I are making sure we have a shared understanding of what's been implemented to date. More context in Slack.

@Ryasmeen: please disregard the question I posed in T291307#7630898 for now.

Before we do any further testing of the bucketing criteria, @DLynch, @MNeisler, and I are making sure we have a shared understanding of what's been implemented to date. More context in Slack.

Alright, after talking with @DLynch and @MNeisler offline [i], I can confirm no changes are needed to the bucketing criteria that's currently been implemented.

I've updated the task description's ===Requirements > Bucketing Criteria section [ii] so that it matches what's been implemented.


i. https://wikimedia.slack.com/archives/G01625LGGQZ/p1642636849000100
ii. T291307#7641776

@Ryasmeen: please disregard the question I posed in T291307#7630898 for now.

Before we do any further testing of the bucketing criteria, @DLynch, @MNeisler, and I are making sure we have a shared understanding of what's been implemented to date. More context in Slack.

Alright, after talking with @DLynch and @MNeisler offline [i], I can confirm no changes are needed to the bucketing criteria that's currently been implemented.

I've updated the task description's ===Requirements > Bucketing Criteria section [ii] so that it matches what's been implemented.


i. https://wikimedia.slack.com/archives/G01625LGGQZ/p1642636849000100
ii. T291307#7641776

@ppelberg: After following the last few comments here and the conversation on the editing-instrumentation channel, it seems we have not changed any bucketing criterias which means we are going to keep the criteria as such that the bucketing will happen for People who have NOT used the New Discussion Tool before, as determined by the state of the discussiontools-newtopictool-opened preference.

Based on that understanding, I have re-tested the A/B test bucketing for New Discussion Tool again and can confirm that people are now being bucketed as per the requirements stated in the task description and after the last two patches all the requirements are met.

So, I think we are good to go here.

...Based on that understanding, I have re-tested the A/B test bucketing for New Discussion Tool again and can confirm that people are now being bucketed as per the requirements stated in the task description and after the last two patches all the requirements are met.

So, I think we are good to go here.

Excellent – thank you for double checking, @Ryasmeen.

Having verified the task description's bucketing ===Requirements have been met, this task can be resolved.

NOTE: we will verify that people are being bucketed into the control and test groups in the proportions we expect in T291308.