Page MenuHomePhabricator

Publish multi-check measurement plan
Open, MediumPublic

Description

In T325838, we wrote [i] and published [ii] the initial measurement plan that is guiding how we are evaluating the initial Reference Edit Check.

This task involves the work of extending that initial measurement plan [i] to include the multi-check experience T347530 will introduce

Process

StepDescriptionPeople ResponsibleStatus
1.Draft measurement plan (e.g. project goals, hypotheses, research questions, experiment plan, etc.)@ppelberg Drafting in progress
2.Review measurement plan draft + propose methods/metrics for evaluating hypotheses and answering research questions@MNeisler + @nayoub
3.Review measurement plan as a teamEditing-team
4.Finalize measurement plan@MNeisler + @ppelberg
5.Publish measurement plan on mediawiki.org@ppelberg

i. Edit Check/Measurement Plan
ii. mw:Edit_check#Evaluating_impact

Event Timeline

MNeisler triaged this task as Medium priority.
MNeisler added a project: Product-Analytics.
MNeisler moved this task from Triage to Current Quarter on the Product-Analytics board.

Per what @MNeisler and I discussed offline today, the experiment(s) we'll run of the initial Multi-Check experience [i] will be meant to help us decide the following...

What is the maximum number of Edit Checks that can be presented within the "Proofreading moment" before editors…?

  • Abandon the changes they've made with or without engaging with the Edit Checks relevant to them
  • Publish low quality edits
  • Are deterred from returning to edit again

Next: @ppelberg to update the Multi-Check Measurement Plan to reflect the above and file a new ticket that includes the "hypotheses" and "curiosities" the experiment(s) we run will seek to offer conclusions around.

Note: some of the aforementioned "hypotheses" and "curiosities" will be borrowed from T342930.


i. Initially, the Multi-Check experience will present people with multiple Reference Checks (when applicable) within the existing "proofreading" moment. Future iterations of the Multi-Check experience will introduce new Edit Checks as well as a new moment within the editing workflow when these Checks will be presented within.