Note: I've updated the task description with the decisions @MNeisler and I came to in the meeting we had yesterday.

• People who have explicitly enabled or disabled the overall Discussion tools beta feature

Isn’t a negation missing here? It’d be strange if someone needed to do something explicitly (toggle the Discussion tools beta feature) in order to participate in the A/B test.

In T304030#7805881, @Tacsipacsi wrote:

• People who have explicitly enabled or disabled the overall Discussion tools beta feature

Isn’t a negation missing here? It’d be strange if someone needed to do something explicitly (toggle the Discussion tools beta feature) in order to participate in the A/B test.

Good spot, @Tacsipacsi. Do you think this change resolves the ambiguity you identified?

Thanks, look good!

TBD: do we include people in the A/B test who have subscribed to a topic at any point prior to the A/B test beginning?

Per the conversation @MNeisler and I had offline today, we are going to INCLUDE people who have used Topic Subscriptions before in the A/B test provided the following two conditions are met:

1. In our analysis, we can differentiate between people who have used Topic Subscriptions prior to the A/B test beginning and those who have not
   • Where "used" in this context means that you have subscribed to at least one topic, be it manually by clicking a [ subscribe ] link or automatically, by way of commenting in a discussion or starting a new one.
2. In our analysis, we can log when people who have used Topic Subscriptions prior to the A/B test beginning re-enable or disable the Enable topic subscription setting within Special:Preferences.

To be clear, "People who have used [i] Topic Subscriptions prior to the A/B test beginning." in the requirements just means also-including them, not only-including them?

1. In our analysis, we can differentiate between people who have used Topic Subscriptions prior to the A/B test beginning and those who have not

I believe that manually checking the discussiontools_subscription table and seeing whether sub_created is before the start would qualify?

2. In our analysis, we can log when people who have used Topic Subscriptions prior to the A/B test beginning re-enable or disable the Enable topic subscription setting within Special:Preferences.

I'm not sure what logging is available on preference-changing. Megan could comment?

In T304030#7904036, @DLynch wrote:

To be clear, "People who have used [i] Topic Subscriptions prior to the A/B test beginning." in the requirements just means also-including them, not only-including them?

Good catch. Yes, people who have and have NOT used Topic Subscriptions prior to the A/B test beginning ought to be included in the test.

1. In our analysis, we can differentiate between people who have used Topic Subscriptions prior to the A/B test beginning and those who have not

I believe that manually checking the discussiontools_subscription table and seeing whether sub_created is before the start would qualify?

@MNeisler confirmed during the meeting we had today that using the sub_created field will work.

2. In our analysis, we can log when people who have used Topic Subscriptions prior to the A/B test beginning re-enable or disable the Enable topic subscription setting within Special:Preferences.

I'm not sure what logging is available on preference-changing. Megan could comment?

Sounds good. Per the conversation @MNeisler and I had offline today, Megan to investigate.

In the meantime, I've updated the task description to clarify that we are wanting to know whether people enable or disable the Enable topic subscription setting within Special:Preferences after the A/B test begins.

In our analysis, we can log when people who have used Topic Subscriptions prior to the A/B test beginning re-enable or disable the Enable topic subscription setting within Special:Preferences.

I'm not sure what logging is available on preference-changing. Megan could comment?

The PrefUpdate table logs changes to user preferences and would include the data needed to determine when and if a user re-enabled or disabled a user preference. However, only properties on the allowlist are logged.

The allowlist currently includes the discussiontools-betaenable property but does not include the topic subscription settings. These would need to be added in order to log this data. The allowlist is modifiable though and I don't think it will be a problem to get these added. I created a ticket to do this in T307733.

Note: PrefUpdate data is sanitized after 90 days (user id info is removed) so we would need to review this metric within 90 days of the start of the AB test.

Change 793359 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Ready A/B test code for topic subscriptions

https://gerrit.wikimedia.org/r/793359

Change 793360 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikiEditor@master] Update schema logging of bucket for new DiscussionTools A/B test

https://gerrit.wikimedia.org/r/793360

Change 793361 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Update schema logging of bucket for new DiscussionTools A/B test

https://gerrit.wikimedia.org/r/793361

Change 793359 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Ready A/B test code for topic subscriptions

https://gerrit.wikimedia.org/r/793359

Change 793361 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Update schema logging of bucket for new DiscussionTools A/B test

https://gerrit.wikimedia.org/r/793361

Change 793360 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Update schema logging of bucket for new DiscussionTools A/B test

https://gerrit.wikimedia.org/r/793360

Looks like the last step here is to make a config change to enable this on the beta cluster (similar to 199b043c73e573b96527eff9907f5c33c6debe74).

Change 798976 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Update beta cluster DiscussionTools A/B test config

https://gerrit.wikimedia.org/r/798976

Change 798976 merged by jenkins-bot:

[operations/mediawiki-config@master] Update beta cluster DiscussionTools A/B test config

https://gerrit.wikimedia.org/r/798976

Mentioned in SAL (#wikimedia-operations) [2022-05-24T21:36:37Z] <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:798976|Update beta cluster DiscussionTools A/B test config (T304030)]] (duration: 00m 49s)

In T304030#7945964, @matmarex wrote:

Looks like the last step here is to make a config change to enable this on the beta cluster (similar to 199b043c73e573b96527eff9907f5c33c6debe74).

@matmarex: is there anything that you think needs to be added to or changed about the steps in the task description's ===Minimal test case section for @Ryasmeen / @EAkinloose to verify the bucketing has been implemented in a way that meets the ===Requirements?

I hope it's clear, the bucketing also works mostly the same as the last A/B test, so you can refer back to that (T291307). Feel free to ask me if you run into any issues.

@matmarex: few questions and observations from my side:

1. On the task description, it mentions after logging into Beta Cluster, "Verify you have been assigned to the test or control group. But, there is no event that fires at that point to check that, unless I click on "Edit Source" ( mwedit.init fires on load) or other talk page related elements on the page such as "Add topic" or "Subscribe" link (when it's present). Just wanted to confirm that, we are not expecting any events to fire when we just load the talk page itself right?

2. I noticed a case where for an existing user, the bucketing group was control but they still got the 'subscribe' links. This user has previously explicitly turned on the "topic subscription" if I remember correctly. Does that mean that for an existing user step 4 can be false? (I would expect so, we probably don't want to turn it off for the purpose of a/b test if they have explicitly turned it on before, but under Meta section @ppelberg mentioned "People who have explicitly enabled or disabled the overall Discussion tools beta feature should NOT be excluded from the A/B test", so I am a bit confused here)

3. I do not have a test user account on Beta cluster that did not enable "topic subscription" prior to the A/B test beginning, so I am not being to check this specific scenario. Does anyone have such accounts?

4. It mentions on the spec that bucketing should be done on a per-Wikipedia basis. Does that mean that the same user might get bucketed into different groups in different wikis.

5. I also noticed if a user has previously bucketed into 'control' group and had no 'Subscribe' links appearing on a talk page, if they then disable and then re-enable the "Discussion Tools" beta feature, the subscribe links start appearing at that point even though the bucketing group didn't change to 'test'.

The following scenarios are working correctly:

1. Logged out users are being excluded from the A/B test.
2. Users are staying in the same group for the duration of the test, even if they explicitly turn on or off the topic subscription.
3. People who have used Topic Subscriptions prior to the A/B test beginning are getting bucketed.
4. Users are staying in the same group for the duration of the test, even if they explicitly turn on or off the Discussion Tools beta feature.

1 On the task description, it mentions after logging into Beta Cluster, "Verify you have been assigned to the test or control group.

You can check via the javascript console: mw.config.get('wgDiscussionToolsABTestBucket')

2. I noticed a case where for an existing user, the bucketing group was control but they still got the 'subscribe' links

Bucketing won't change anything about whether someone has manually enabled the feature already; it (effectively) changes the default behavior, rather than meddling in the user's settings.

4. It mentions on the spec that bucketing should be done on a per-Wikipedia basis. Does that mean that the same user might get bucketed into different groups in different wikis.

Yes. The bucketing is based on the user ID, and (as I understand it) the "same user" is actually a big pile of synchronized user-accounts that're automatically created on other wikis via the CentralAuth extension, and so which aren't guaranteed to have the same user ID.

5. I also noticed if a user has previously bucketed into 'control' group and had no 'Subscribe' links appearing on a talk page, if they then disable and then re-enable the "Discussion Tools" beta feature, the subscribe links start appearing at that point even though the bucketing group didn't change to 'test'.

The bucket shouldn't change in this situation. It's plausible that this behavior-change might just be from hitting save in the preferences cleaning up some changed defaults -- I'd have to investigate further to know exactly what happened.

In T304030#7972647, @DLynch wrote:

1 On the task description, it mentions after logging into Beta Cluster, "Verify you have been assigned to the test or control group.

You can check via the javascript console: mw.config.get('wgDiscussionToolsABTestBucket')

Right, totally forgot about this! Okay, so that's also working as expected.

In T304030#7972647, @DLynch wrote:

2. I noticed a case where for an existing user, the bucketing group was control but they still got the 'subscribe' links

Bucketing won't change anything about whether someone has manually enabled the feature already; it (effectively) changes the default behavior, rather than meddling in the user's settings.

@Ryasmeen, in addition to what @DLynch mentioned above, it would be accurate for you to think, "...that for an existing user step 4 [in the task description's ===Minimal test case section) can be false?"

Note: with @MNeisler confirming the sub_created field will be sufficient for us to differentiate between people who used Topic Subscriptions prior to the A/B test beginning and those who did not.

In T304030#7972644, @Ryasmeen wrote:

@matmarex: few questions and observations from my side:

3. I do not have a test user account on Beta cluster that did not enable "topic subscription" prior to the A/B test beginning, so I am not being to check this specific scenario. Does anyone have such accounts?

I do not have such an account..

@EAkinloose do you happen to have a test user account on Beta cluster that did not enable "topic subscription" prior to the A/B test beginning on 24 May that would enable you to test the scenario @Ryasmeen is describing above?

In T304030#7972647, @DLynch wrote:

5. I also noticed if a user has previously bucketed into 'control' group and had no 'Subscribe' links appearing on a talk page, if they then disable and then re-enable the "Discussion Tools" beta feature, the subscribe links start appearing at that point even though the bucketing group didn't change to 'test'.

The bucket shouldn't change in this situation. It's plausible that this behavior-change might just be from hitting save in the preferences cleaning up some changed defaults -- I'd have to investigate further to know exactly what happened.

@DLynch / @Ryasmeen: what would you name the task where we would potentially do the investigation being described above? I ask this in an effort to better understand the issue y'all are talking about.

In T304030#7974948, @ppelberg wrote:

In T304030#7972647, @DLynch wrote:

2. I noticed a case where for an existing user, the bucketing group was control but they still got the 'subscribe' links

Bucketing won't change anything about whether someone has manually enabled the feature already; it (effectively) changes the default behavior, rather than meddling in the user's settings.

@Ryasmeen, in addition to what @DLynch mentioned above, it would be accurate for you to think, "...that for an existing user step 4 [in the task description's ===Minimal test case section) can be false?"

Note: with @MNeisler confirming the sub_created field will be sufficient for us to differentiate between people who used Topic Subscriptions prior to the A/B test beginning and those who did not.

Understood.

The bucket shouldn't change in this situation. It's plausible that this behavior-change might just be from hitting save in the preferences cleaning up some changed defaults -- I'd have to investigate further to know exactly what happened.

@DLynch / @Ryasmeen: what would you name the task where we would potentially do the investigation being described above? I ask this in an effort to better understand the issue y'all are talking about.

Based on the discussion we had on our team channel, we concluded that this is an expected behavior because when someone disables the "Discussion Tools" beta feature it clears out the settings in preferences and when they re-enable it, it restores the defaults which means "topic subscription" will be in enabled state and since user's settings has more preference, it wins over the a/b test group settings.

So @ppelberg, I don't think we need to create any other task for this. (@matmarex correct me if I mis-phrased something here.)