Conduct pre-mortem for Edit Check project
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Dec 22 2022, 10:51 PM

Description

This task is about identifying, as a team, the potential ways in which the Edit Check project could fail or otherwise not deliver the impact we're implementing it to have.

We will also use this task to determine how we will identify, and respond to, a scenario were it to arise.

Pre-mortem Link

Edit Check/Pre-Mortem

Process

Step	Description	Person/People Responsible	Status	Format
1.	Identify and document potential failure scenarios	Editing-team	Done	Async
2.	Cluster failure scenarios	@ppelberg	Done	Async
3.	Estimate impact and likelihood of each failure scenario	Editing-team	Done	Sync/Async
4.	Prioritize failure scenarios to plan for	Editing-team	Done	Async
5.	Propose leading indicators (e.g. qualitative and quantitative) for prioritized scenario	@MNeisler	Done	Async
6.	Propose a plan of action/mitigation strategy for each `High` and `Medium` Priority scenario	Editing-team	Done
7.	Relate each scenario to a planned analysis that will be defined in the experiment plan	@MNeisler	Done	Async

Related Objects
Search...

Status	Assigned	Task
Open	None	T265163 Create a system to encode best practices into editing experiences
In Progress	Trizek-WMF	T331946 [RELEASE TICKET] Make Edit Check (references) available to all newcomers at all Wikipedias
Open	None	T331947 [RELEASE TICKET] Make Edit Check MVP available to all newcomers at Partner Wikipedias
Resolved	ppelberg	T338907 [RELEASE TICKET] Make Edit Check MVP available as Beta Feature (all wikis)
Resolved	None	T332646 [Deployment] Make Edit Check MVP (mobile + desktop) available for testing at test.wikipedia.org
Resolved	ppelberg	T325851 Conduct pre-mortem for Edit Check project

Event Timeline

ppelberg created this task.Dec 22 2022, 10:51 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 22 2022, 10:51 PM

ppelberg moved this task from Untriaged to Upcoming on the Editing-team board.Dec 22 2022, 10:51 PM

ppelberg moved this task from To Triage to Triaged on the VisualEditor board.

ppelberg added a parent task: T265163: Create a system to encode best practices into editing experiences.

ppelberg moved this task from Upcoming to Needs Discussion / Investigation on the Editing-team board.Jan 9 2023, 5:27 PM

VPuffetMichel added a project: EditCheck.Jan 18 2023, 3:58 PM

ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team.Feb 6 2023, 5:54 PM

• JMcLeod_WMF claimed this task.Feb 15 2023, 7:52 PM

ppelberg triaged this task as High priority.Feb 22 2023, 5:15 PM

ppelberg moved this task from Incoming to Doing on the Editing-team (Kanban Board) board.Mar 2 2023, 7:53 PM

Update: I've clustered the the failure scenarios that surfaced during the 1 March 2023 meeting the Editing Team had offline.

See: Edit Check/Pre-Mortem --> Failure Scenarios

• JMcLeod_WMF reassigned this task from • JMcLeod_WMF to ppelberg.Mar 30 2023, 1:50 PM

• JMcLeod_WMF subscribed.

In T325851#8700640, @ppelberg wrote:

Update: I've clustered the the failure scenarios that surfaced during the 1 March 2023 meeting the Editing Team had offline.

See: Edit Check/Pre-Mortem --> Failure Scenarios

The Failure scenarios tab of the Edit Check/Pre-Mortem sheet now reflects all of failure scenarios the team identified in the 1 March meeting I referenced in T325851#8700640 as well as the scenarios that were contributed asynchronously, before that meeting.

NOTE: I've updated Person/People Responsible column within the task description to reflect the fact that I will be away from the wikis (E1630) while work on the remaining steps takes place.

ppelberg edited parent tasks, added: T332646: [Deployment] Make Edit Check MVP (mobile + desktop) available for testing at test.wikipedia.org; removed: T265163: Create a system to encode best practices into editing experiences.Apr 1 2023, 8:18 AM

ppelberg updated the task description. (Show Details)

MNeisler updated the task description. (Show Details)May 10 2023, 4:50 PM

MNeisler updated the task description. (Show Details)May 11 2023, 4:18 PM

MNeisler updated the task description. (Show Details)

Update:

The Editing team has estimated the impact and likelihood of each failure scenario. All the scenarios in the Edit Check/Pre-Mortem sheet now have a priority.

I've proposed leading indicators (e.g. qualitative and quantitative) for each prioritized scenario and related each scenario to the final measurement plan. This included (1) adding a few new metrics to the measurement plan to account for any leading indicators not already identified and (2) also adding a note to each leading indicator in the pre-mortem sheet to identify the related experiment/analysis.

Next Steps:
The Editing team will review the prioritized leading indicators with @ppelberg once he's back and confirm a plan of action/mitigation strategy.

MNeisler updated the task description. (Show Details)May 11 2023, 8:11 PM

ppelberg moved this task from Doing to Ready for Sign Off on the Editing-team (Kanban Board) board.Jul 26 2023, 12:25 AM

ppelberg updated the task description. (Show Details)Aug 22 2023, 9:48 PM

ppelberg updated the task description. (Show Details)

Propose a plan of action/mitigation strategy for each High and Medium Priority scenario

This is done; we've incorporated the "High" and "Medium" priority scenarios into the measurement plan by way of drafting a set of guardrails which can now be seen on-wiki in the Undesirable Outcomes table here: https://www.mediawiki.org/wiki/Edit_check#Evaluating_impact.

ppelberg mentioned this in T342930: [MILESTONE] Run an A/B test to evaluate Edit Check (references) impact.Oct 18 2023, 10:16 PM

ppelberg mentioned this in T352121: Conduct Pre-mortem for multi-check experience.Nov 28 2023, 12:29 AM

Conduct pre-mortem for Edit Check projectClosed, ResolvedPublicActions