Page MenuHomePhabricator

Implement experiment "DE1.2.2 Motivational moments A/A test"
Open, Needs TriagePublic

Description

Instrumentation specification

https://docs.google.com/spreadsheets/d/1Ywz2_KlhtjZrqcH5I_ZHiJ7owMD2JtTCgF_u-8xelpY/edit?gid=0#gid=0

Acceptance criteria

  • Experiment is implemented in code
    • Users in Control (control) and Control 2 (control-2) groups get status quo (default) experience
    • Experiment tracking key and variation keys are aligned with the values in the linked instrumentation spec
  • Instrumentation spec is implemented and tested
    • experiment_exposure events sent when experiment subject visits their Collections (since that's where we will have the upcoming A/B test of signals)
    • All events in the spec are marked Yes (done)
    • All specified events fire in relevant scenarios and user interactions
    • All events data appears to be accurate and includes all the specified interaction data
  • Experiment is configured in GrowthBook: https://growthbook.wikimedia.org/experiment/exp_anu814nmufxo852#overview
  • Create a Phab task to clean up experiment code after the experiment is over: T439048: [DE1.2.2] Clean up A/A test code

Related Objects

Event Timeline

Instrumenting Average contributor engagement count for this experiment is currently blocked on:

Soft dependency on T437414: editSaved generic instrumentation doesn't handle edits made through other editors because it would be good to have more comprehensive edit_saved event coverage.

@VPuffetMichel: I would consider T441339: Integrate updated postEdit hook into Ask your mentor a question a soft-block (not critical dependency). It would be GREAT if we had those for the A/A, but it's totally OK if we don't and instead just have it eventually. There are a variety of editing surfaces (some owned by Editing, some owned by Growth) that will need to be audited & updated once ExPlat is done with T437414. "Ask your mentor a question" is one of those surfaces. None of that is blocking this.

I think Connection is clear to pick this up and start working on it.

Following up to @OBenhmida's questions:

Our A/B will run on personal lists on Collections, so a few questions:

  • Enrollment is “any page visit”. Is that logged-in users only? Our real tests will target editors by edit count (3+ for signals, under 5 for a “Before you go” prompt), so should the A/A match that, and is performer_edit_count collected?

The A/A is currently configured to be logged-in experiment, yes. https://growthbook.wikimedia.org/experiment/exp_anu814nmufxo852#overview

This includes permanent and temporary users. Do you want to exclude temporary users?

In terms of edit count-based targeting, let's keep it simpler for the A/A. But that's good to know that's your plan for the real tests – those would need to document their eligibility criteria explicitly and implement those checks in code.

We do automatically collect performer_edit_count with exposure events, because that powers the various edit count-based dimensions we offer in GrowthBook for breaking down experiment results across segments.

  • create_worklist is listed under the engagement count metric. Can we keep it out of the KR metric and track it separately? Otherwise the treatment arm gets credit just for using the feature.

That's a good point. Sure, let's keep it out of the metric definition for now and only add it if the feature goes out to everyone. (DONE)

  • Does the GrowthBook metric exclude thanks received, per our definition?

GrowthBook metric includes both thanks given and thanks received. (Same with WikiLove.)

  • Our main hypothesis is scored on return frequency (weeks active out of 8). Can we add that as a metric using the same events?

Not exactly – what we can add is a daily participation rate type metric such as Daily web visit rate.

When you say "return frequency" is that a return to a specific Special page or just the website in general?

Thanks @mpopov, this is really helpful.

Temporary users: Yes, let’s exclude them. Our tests will target editors by edit count on personal lists, so the A/A should match that population.

Edit count targeting: Agreed, simpler for the A/A. We’ll document eligibility for each A/B (3+ edits for signals, 1 to 99 edits for the “Before you go” prompt) and implement the checks in code.

create_worklist: Thank you.

Thanks and WikiLove received: Our engagement definition counts actions the person takes, so I’d like to drop thanks_received and wikilove_received from this metric. Two reasons: in an A/B, a control user thanked by a treatment user would pick up credit for the treatment’s effect, and each thank would be counted twice. We’d still like the received events as a separate secondary metric, since recognition received matters to us. Does that work on your side?

Return frequency: Neither, actually. We mean coming back to contribute: the share of days (or weeks) in which someone did at least one of the actions in the engagement metric. Visits would mostly capture reading. Could we have a participation rate built on those same events? Visits to a person’s Collection would be a useful feature metric for the A/B, but not the main one.

We’ll get the implementation assigned so we’re ready to walk through it when you’re back.

Temporary users: Yes, let’s exclude them. Our tests will target editors by edit count on personal lists, so the A/A should match that population.

Got it – eligibility criteria noted in T439045: Experiment: DE1.2.2 Motivational moments A/A test

Thanks and WikiLove received: Our engagement definition counts actions the person takes, so I’d like to drop thanks_received and wikilove_received from this metric. Two reasons: in an A/B, a control user thanked by a treatment user would pick up credit for the treatment’s effect, and each thank would be counted twice. We’d still like the received events as a separate secondary metric, since recognition received matters to us. Does that work on your side?

We can track thanks received as a separate metric, sure. Changed:

When you say "return frequency" is that a return to a specific Special page or just the website in general?

Return frequency: Neither, actually. We mean coming back to contribute: the share of days (or weeks) in which someone did at least one of the actions in the engagement metric. Visits would mostly capture reading. Could we have a participation rate built on those same events? Visits to a person’s Collection would be a useful feature metric for the A/B, but not the main one.

Hm… Might need to discuss this in the DE1.2 channel.

When you say "return frequency" is that a return to a specific Special page or just the website in general?

Return frequency: Neither, actually. We mean coming back to contribute: the share of days (or weeks) in which someone did at least one of the actions in the engagement metric. Visits would mostly capture reading. Could we have a participation rate built on those same events? Visits to a person’s Collection would be a useful feature metric for the A/B, but not the main one.

Hm… Might need to discuss this in the DE1.2 channel.

Sorry, I reread your comment more carefully and yeah we can totally have that. Done: Daily contributor engagement rate

experiment_exposure events sent when experiment subject visits their Collections (since that's where we will have the upcoming A/B test of signals)

Or a Collection, technically? Because the signals are only going to be shown if somebody is looking at a specific Collection, right?