Page MenuHomePhabricator

Placeholder: Set up test kitchen for Event Pathways (part 1)
Open, Needs TriagePublic

Description

Acceptance Criteria:

  • Set up so we can track the following:
    • Test A (if an editor with make an edit to an article that matches articles in the worklist of an upcoming event on the wiki where they are editing, show post-edit event invitation):
      • Priority 1: Click through rate to event page
      • Priority 2: Do they register for the event?
      • Priority 3: Do they make any edits in the event via the collaborative contributions?

Event Timeline

@ifried quick question: you mentioned before that you would want to instrument to track adoption and health outcomes, more of a "before and after" than a true experiment. Is that still the case, or will the team be running an experiment? The reason I ask is that I expect sample size to be quite limiting given the criteria. Happy to talk through this!

@JVanderhoop-WMF: Hey there! Yes, we're interested in click-through-rates and health outcomes, rather than running an A/B test. We have a few reasons why, but the main reason that we don't want to do an A/B test for this work, most likely, is that we just don't think we would learn very much from it (for this particular set of features). We are already pretty confident that any control group would *not* be able to find the events we promote. So, it would be extra set-up for minimal learning benefits.

As for the sample size, yes, our experiment A will likely have a very small sample size. However, we're now considering launching an experiment B, which could have a much larger sample size (since it could conceivably target any newcomer or junior editor who makes a reasonably substantial edit to an article on any wiki). You can see some of our ideas for experiment B in this ticket. The sample size of experiment B could be so large that we may want to limit to certain pilot wikis. We're still finalizing exactly what will be experiment B (it may change), but the fundamental idea is that we reach out to a lot of junior/newcomer editors if they make edits in an article that is merely within the same topical area of the event. There are only a few topics within LiftWing, so the likelihood of promoting an event to them is much higher than if there is a worklist match.

Ok great - thanks! In that case, your team will want to follow steps for measuring product health

With this, you won't be running an experiment, but you will be using Test Kitchen. (we help with both!)

ifried renamed this task from Investigation: Determine if and how we can get necessary data from test kitchen for event discovery expeirments to Investigation: Determine if and how we can get necessary data from test kitchen for event discovery experiments.Fri, May 22, 8:04 PM
ifried updated the task description. (Show Details)

Yes, @JVanderhoop-WMF, thanks for the clarification and link to documentation. Not an experiment, but running Test Kitchen. Much appreciated!

ifried renamed this task from Investigation: Determine if and how we can get necessary data from test kitchen for event discovery experiments to Placeholder: Set up test kitchen for Event Pathways (part 1).Mon, Jun 8, 6:45 PM
ifried updated the task description. (Show Details)
ifried added a subscriber: cmelo.

@cmelo, can you add some technical details to the task for how we set this up? Thank you!