Page MenuHomePhabricator

Analyze impact of edit cards in the mobile visual editor
Closed, ResolvedPublic

Description

Hypotheses and metrics

Edit Cards and the changes to the subsequent dialogs will:

hypothesis 1: ...make contributors more likely to start adding and modifying links and citations, since the actions to edit them will be more prominent on the page
metric 1: average number of link and citation workflow starts per session

hypothesis 2: ...make contributors more likely to finish adding and modifying links and citations, since the steps to do so will be more visible and clear.
metric 2: link and citation workflow completion rate

hypothesis 3: ...make contributors more likely to publish their edits, since it will be more clear when they have finished their task.
metric 3: edit completion rate

Instrumentation

Based on our post-deployment checks (T221254), we have detailed data on the linking and citation workflows starting 13 July. For details about the linking and citation workflows, see the feature instrumentation spec and the data dictionary.

Experiment plan

We will do a basic pre/post analysis of whether the metrics above changed due to the multiple iterative deployments of Edit Cards:

versiondate deployed
v121-June to Bengali, Hebrew, and Persian Wikipedias (T221314#5272510); v1 was never deployed to all wikis
v21-August 2019 in 1.34.0-wmf.16 (T225834)
v329-August 2019 in 1.34.0-wmf.20 (T229830)
v412-September-2019 in 1.34.0-wmf.22 (T231342)

We should exclude any sessions that have a bucket set in EditAttemptStep, because those are part of the mobile-VE-as-default A/B test.

Glossary

  • Edit cards: the proposed new design of the Context items on mobile. These "edit cards" will show details about, and actions related to, different elements within articles, like links, citations, images, infoboxes, templates, etc. (Developer Glossary)
  • Context Item: a pop-up that appears anytime the cursor or focus is within the context of a specific node or annotation. It shows information, but doesn’t let you edit directly. Multiple context items can appear at once as a stack when one cursor position is in multiple contexts. (Developer Glossary)
    • This is the pre-Edit Card implementation of mobile context items (i.e. the shelf that appears beneath the toolbar in mobile VE)
  • Abort: a contributor has “discarded” their changes/exited the dialog
  • Finish: a contributor has “applied” their changes after making changes to a dialog.
    • Example: a contributor has successfully changed the label of an existing link.
  • Publish: a contributor has made changes and successfully publishes those changes (i.e. they complete their edit(s)).
    • Example: Adds new wiki link > writes edit summary > publishes edit.
  • Changes: whether or not a contributor has typed in the dialog and to a rough extent, how much they’ve been typing

Related Objects

Event Timeline

JTannerWMF added a subscriber: JTannerWMF.

Waiting on V2 to go live

Our Edit Cards analysis is intended to come to conclusions about the following "Hypotheses" and answer the following "Research questions."

The below are copied over from this task: T221246

Something else to note, @Neil_P._Quinn_WMF raised the question below, which our work on this task depends on having the answer to:

  • Confirm: Do we have data on the pre-Edit Card state of VE to compare the post-Edit Cards v3 state of mobile VE to?
    • @Neil_P._Quinn_WMF, this comment leads me to think that "Yes, we do." but leaving to you to confirm: T221254#5376056

Hypotheses

  • Contributors are more likely to start modifying links and citations if the actions to edit them are more prominent/visible on the page
  • Contributors are more likely to finish adding and/or modifying links and citations if the steps to do so are more visible and clear
  • Contributors are more likely to publish their edits when it is more clear when they have finished their task

Research questions

  • Do Edit Cards and the changes to the subsequent dialogs cause contributors to start modifying existing links and citations more often compared to the pre-Edit Card state of VE?
  • Do Edit Cards and the changes to the subsequent dialogs cause a higher percentage of contributors to finish [1] adding and modifying links and citations compared to the pre-Edit Card state of VE?
  • Do Edit Cards and the changes to the subsequent dialogs cause a higher percentage of contributors to save/publish their edits compared to the pre-Edit Card state of VE?

  1. "Finish": Contributors complete the workflow. It does not mean they save their edit(s).

⚠️ Note: I will add the contents of this comment to the task description once access is granted from above.

I just consolidated our different experiment plans into the task description here.

Something else to note, @Neil_P._Quinn_WMF raised the question below, which our work on this task depends on having the answer to:

  • Confirm: Do we have data on the pre-Edit Card state of VE to compare the post-Edit Cards v3 state of mobile VE to?
    • @Neil_P._Quinn_WMF, this comment leads me to think that "Yes, we do." but leaving to you to confirm: T221254#5376056

We have detailed data on linking and citations starting on 13 July (I just added this to the description). However, I could not figure out the deployment dates for the various versions, so I'm not sure if that's before our first Edit Cards deployment. @Esanders, @DLynch, could you help filling in the table of deployment dates in the description?

nshahquinn-wmf renamed this task from Analyze impact of mobile visual editor structured workflows to Analyze impact of edit cards in the mobile visual editor.Sep 23 2019, 5:50 PM
In T221728#5471778, @Neil_P._Quinn_WMF wrote:

I just consolidated our different experiment plans into the task description here.

Something else to note, @Neil_P._Quinn_WMF raised the question below, which our work on this task depends on having the answer to:

  • Confirm: Do we have data on the pre-Edit Card state of VE to compare the post-Edit Cards v3 state of mobile VE to?
    • @Neil_P._Quinn_WMF, this comment leads me to think that "Yes, we do." but leaving to you to confirm: T221254#5376056

We have detailed data on linking and citations starting on 13 July (I just added this to the description). However, I could not figure out the deployment dates for the various versions, so I'm not sure if that's before our first Edit Cards deployment. @Esanders, @DLynch, could you help filling in the table of deployment dates in the description?

Updating the task description with the dates @Esanders was able to uncover for V2 and V3.

It looks like we're still in need of dates for the following:

  • When was v1 live on all wikis?
  • When was v4 live on all wikis?

When was v1 live on all wikis?

Never - we removed the feature flag at the same time as v2.

When was v4 live on all wikis?

We did some fixes after Wikimania but I don't know if they all went in the same release. Do you have a list of which tasks you considered "v4"?

LGoto triaged this task as Medium priority.Oct 21 2019, 5:07 PM

When was v1 live on all wikis?

Never - we removed the feature flag at the same time as v2.

Right, okay. Updating the task description to reflect this.

When was v4 live on all wikis?

We did some fixes after Wikimania but I don't know if they all went in the same release. Do you have a list of which tasks you considered "v4"?

It looks like v4 [1] ended up including the following tasks:

@Esanders, assuming the tasks in the table above make up "Edit Cards v4" and considering T229841 was deployed on 10-Sep, does this mean we call the release date for Edit Cards v4: 12-September-2019 in 1.34.0-wmf.22?


  1. T231342

Updating task description after confirming the details in T221728#5618122 with Ed.

I completed the pre and post-deployment analysis of edit cards on mobile VE.

I'll follow-up with a high-level summary of observations and takeaways but in the meantime here's a link to the current report and repo.

Quick summary of key observations below. See full report and repo for details and charts.

Note: Users in the VE as default AB test were removed during data analysis to help isolate impacts to edit card changes.

  • Edit cards did not strongly impact the efficacy of mobile VE users.
    • Following v4 deployment, the average link and citation workflow starts per session have been fairly stable and the workflow completion rates decreased slightly.
    • Note: Between v3 and v4 deployments (around September 4, 2019), there was a sharp decrease in the daily average number of link workflow starts per session for the window-open-from-tool action (recorded when a card was opened by clicking a button in the toolbar). Further investigation may be needed to determine if this is caused by one of the changes deployed around this time leading to change in user behavior or how events are recorded.

average_link_workflow_starts_byaction.png (1×2 px, 246 KB)

  • Edit Card changes increased the overall usage of link and citation edit cards by mobile VE users.
    • Edit card changes increased the overall rate of link and citation workflow starts indicating an increase in the number of Mobile VE contributors using these tools. From v2 to v3 of the edit cards, there was a 13% increase in the overall number of link workflow starts per day and 28.8% increase in citation workflow starts per day. The number of workflow starts has become more stable following v4 deployment but continues to increase slightly for both citations and links.

total_link_workflow_starts_byaction.png (1×2 px, 223 KB)
total_citation_workflow_starts.png (1×2 px, 245 KB)

  • There was also a slight increase in the proportion of sessions with edit card views that begin a link or citation edit. Following v4 deployment, about 13%-18% of sessions that view a link edit card and about 12 to 22% of sessions that view a citation edit card start an edit.
  • Overall Edit Completion Rate. The overall edit completion rate on mobile visual editor has been steadily increasing; however, this trend has been occurring prior to the edit card deployment and may be due to other variables such an increase in the number of new and returning active editors and seasonal changes.
    • There was a noticeable difference in edit completion rate changes across different user experience levels. Following the deployment of edit cards v2 to all wikis, we observed the highest increase in edit completion rates for junior contributors. From August to September 2019, there was an 8.6% increase for contributors with 1-4 edits and 5.8% for contributors with 5 to 99 edits. In contrast, there was a minimal change for contributors with over 100 edits.

edit_completion_rate_overall.png (1×2 px, 181 KB)

@ppelberg OK to resolve this?

Would it disrupt y'alls process if we left this open until I've had the chance to review and resolve any open questions? I plan for this to happen before the end of this week.

In asking the above, I don't mean to suggest that I have a strong preference for closing this task or leaving it open...I'm just wanting to make sure I know where to represent any resulting questions.

@ppelberg we can keep it open for now, it's categorized as "In Review".

@ppelberg this came up again in our grooming session, please let me know when we can resolve!

@ppelberg this came up again in our grooming session, please let me know when we can resolve!

Eeek 🙈

...yes, please feel free to close. If any follow up questions emerge, I will first talk about them with @Mayakp.wiki after which point we can decide if any further investigation/clarification is needed.