Specifically whatever metric we use shouldn't collide with anything that's set by the reply tool. E.g. "discussiontools-editmode preference is empty" would get us only people who had never used either the new discussion or reply tools.

We'd prefer if we could find a trigger that'd ignore usage of the reply tool.

(@DLynch I've updated the task description with the nuanced you shared in T291536#7372792)

As discussed with @ppelberg, instead of excluding people that have used the new discussion tool before, I think it would be more beneficial if we can include and distinguish these users in the test. However, this will still require us to be able to find a reliable way to differentiate prior reply tool users vs prior discussion tool users.

Rationale:

• This would help ensure that our sampling criteria are not overly restrictive and increase the overall number of events we have available to review in the AB test. I think this will be especially important for the case of the New Discussion Tool, which is likely to be used less frequently than the Reply Tool.
• It would provide an additional and valuable data point to understand and differentiate between new and previous users of the tool.
• If we can differentiate between prior users of the reply and the new discussion tool, then I will be able to account for that in AB test analysis and then there is no reason to exclude.

I'm going to go ahead and make an initial attempt at reframing the task description to reflect this new need and proposed solution but please revise as needed or let me know if you have any questions/concerns.

cc @DLynch

I was thinking that a way to sidestep the whole "is there something that we can use to tell someone has used the discussion tool before?" issue would be for us to actually add a new preference value for it. Just store discussiontools-new-discussion-used = 1 the instant someone triggers the tool. I raise this here because if we wanted to take that approach we'd need to act on it a few weeks in advance of starting the A/B test so that the data would actually exist (since starting-a-new-topic is presumably not the most common act.

Since we're talking about adding a first-run experience for the new-discussion, this would probably be the same preference as it'd need to know whether or not to show that anyway.

@DLynch two clarifying questions for you in response to what you shared in T291536#7424732...

I was thinking that a way to sidestep the whole "is there something that we can use to tell someone has used the discussion tool before?" issue would be for us to actually add a new preference value for it.
Just store discussiontools-new-discussion-used = 1...

• Are you able to share what would be required for @MNeisler to relate the discussiontools-new-discussion-used = 1 preference to EditAttemptStep?

...the instant someone triggers the tool.

• Would it be accurate for me to understand "triggers the tool" in this context as meaning: the moment someone clicks an Add topic / New section affordance and the New Discussion Tool begins to open? If so, this sounds good.

Are you able to share what would be required for @MNeisler to relate the discussiontools-new-discussion-used = 1 preference to EditAttemptStep?

Once it exists, we'd enroll people in the a/b test based on that preference not being set. Megan would then be able to tell whether this had been the case based on the bucket field included in EditAttemptStep events. If she wanted to check it out before the a/b test is activated for recent edit sessions to see how much traffic would be affected, she'd presumably be able to look up values in user_properties based on the user_ids found in EditAttemptStep.

Would it be accurate for me to understand "triggers the tool" in this context as meaning: the moment someone clicks an Add topic / New section affordance and the New Discussion Tool begins to open? If so, this sounds good.

That's what I intended, yes. Technically we could have it mean almost anything -- "has dismissed the first-run popup" for instance.

In T291536#7437269, @DLynch wrote:

Thank you for clarifying, @DLynch. Some comments and questions in response below...

Are you able to share what would be required for @MNeisler to relate the discussiontools-new-discussion-used = 1 preference to EditAttemptStep?

Once it exists, we'd enroll people in the a/b test based on that preference not being set.

Question: is it possible for us to include people who have the New Discussion Tool before in the A/B test while also being able to distinguish these people form those who have NOT used the New Discussion Tool before the A/B test began?

Reason I ask: we'd like to include people who have used the New Discussion Tool before in the A/B test while still being able to distinguish these people from those who have NOT used the New Discussion Tool before the A/B test began. More context in T291536#7392780.

Question: considering the above, can you share how – if at all – what we implement to satisfy this ticket relates to the requirement in both T270788 and T293962 that specifies People should only experience the "first-run" experience we are designing and implementing in this ticket once, across wikis ?

Megan would then be able to tell whether this had been the case based on the bucket field included in EditAttemptStep events. If she wanted to check it out before the a/b test is activated for recent edit sessions to see how much traffic would be affected, she'd presumably be able to look up values in user_properties based on the user_ids found in EditAttemptStep.

Comment: noted. During the meeting @MNeisler and I had today, Megan confirmed what you described meets the requirements she has.

Would it be accurate for me to understand "triggers the tool" in this context as meaning: the moment someone clicks an Add topic / New section affordance and the New Discussion Tool begins to open? If so, this sounds good.

That's what I intended, yes. Technically we could have it mean almost anything -- "has dismissed the first-run popup" for instance.

Comment: understood. Let's continue defining "people who have used the New Discussion Tool" in this context as people who have clicked an affordance that causes the New Discussion Tool to open. I've updated the task description to reflect this definition.

Question: is it possible for us to include people who have the New Discussion Tool before in the A/B test while also being able to distinguish these people form those who have NOT used the New Discussion Tool before the A/B test began?

We could give them a different bucket-name, I suppose. Our existing code is just test and control, but there's nothing stopping us from splitting it up further.

Question: considering the above, can you share how – if at all – what we implement to satisfy this ticket relates to the requirement in both T270788 and T293962 that specifies People should only experience the "first-run" experience we are designing and implementing in this ticket once, across wikis ?

It probably should only relate to T293962 -- since that's a feature that'll live inside the new discussion tool, gating anything off the "feature peek" wouldn't make sense. If that preference exists for the "how to go back" hint, we can use that as a good proxy for has-used-new-section-tool.

We can also write a preference that literally just means that without having to infer anything, if we want to. I just like piggybacking off of a "real" preference with a non-analytics purpose if at all possible for efficiency's sake. 😁

In T291536#7449048, @DLynch wrote:

Question: is it possible for us to include people who have the New Discussion Tool before in the A/B test while also being able to distinguish these people form those who have NOT used the New Discussion Tool before the A/B test began?

We could give them a different bucket-name, I suppose. Our existing code is just test and control, but there's nothing stopping us from splitting it up further.

Gotcha, okay.

Question: considering the above, can you share how – if at all – what we implement to satisfy this ticket relates to the requirement in both T270788 and T293962 that specifies People should only experience the "first-run" experience we are designing and implementing in this ticket once, across wikis ?

It probably should only relate to T293962 -- since that's a feature that'll live inside the new discussion tool, gating anything off the "feature peak" wouldn't make sense.

Ah, okay. Understood.

If that preference exists for the "how to go back" hint, we can use that as a good proxy for has-used-new-section-tool.

We can also write a preference that literally just means that without having to infer anything, if we want to. I just like piggybacking off of a "real" preference with a non-analytics purpose if at all possible for efficiency's sake. 😁

Alright, it seems like we're at the point of being ready to converge on an approach to implement. With this in mind, I've:

• ADDED – what I understand to be – the answer to question #1 in the task description's ===Open question section
• CREATED T294167 to do the actual implementation [i]

i. I considered repurposing this ticket for the implementation, but thought it might be easier to leave this as a "divergent" space