Page MenuHomePhabricator

Measure the impact of the wikitext editing tools on the iOS app
Closed, ResolvedPublic

Description

In v6.2, we added some wikitext editing tools to the iOS app (T209168). We also verified that the MobileWikiAppEdit schema is tracking the basic user editing behaviors correctly (T215329).

The purpose of this task is to use data from MobileWikiAppEdit and Mediawiki databases to measure the impact of these new editing tools. Specifically, we're going to look into:

Annual plan metrics:

  • Number of edits
  • Edits/editor
  • Revert rate
  • User retention rate

Other user behavior questions similar to T203723:

  • The count of edits by languages, devices (iPhone vs iPad), log in status (anonymous or not)
  • User funnel: How many/what percent of users start the description editing process? How many/what percent of them make it to the point of saving the edit?

@JMinor and @cmadeo , please let me know if there is anything else you want to know, and what was the intended impact (i.e. what do you expect to see, and expect not to see).

Given that the release date is Feb 21 2019, we want to have at least 30 days data for this task.

Event Timeline

chelsyx moved this task from Triage to Next Up on the Product-Analytics board.
chelsyx updated the task description. (Show Details)Mar 12 2019, 10:45 PM
chelsyx updated the task description. (Show Details)Mar 13 2019, 12:46 AM

@chelsyx, would it be possible to also track where in the editing funnel we're potentially seeing drop off? Eg. Would it be possible to track how many people open the editor and drop off on the first Edit screen, or drop off at the Preview screen, or drop off at the Edit summary screen?

Additionally it'd be great to see which elements on the toolbar users are utilizing.

Thanks! Also sorry if these are elements that are tracked in some other way and I'm misguided!

@chelsyx, would it be possible to also track where in the editing funnel we're potentially seeing drop off? Eg. Would it be possible to track how many people open the editor and drop off on the first Edit screen, or drop off at the Preview screen, or drop off at the Edit summary screen?

@cmadeo With MobileWikiAppEdit table, we can calculate how many people open the editor, drop off before preview, drop off before trying to save, and save successfully. But we can't know how many drop off at the Edit summary screen. Would that work?

Additionally it'd be great to see which elements on the toolbar users are utilizing.

Sorry, we didn't instrument for the elements on the toolbar. How important this kind of information is? Perhaps we can add some tracking in the next release.

@chelsyx thanks for the reply!

@cmadeo With MobileWikiAppEdit table, we can calculate how many people open the editor, drop off before preview, drop off before trying to save, and save successfully. But we can't know how many drop off at the Edit summary screen. Would that work?

That should work! So we can extrapolate who dropped off in the Preview and Edit summary screens by looking at the difference between who dropped after preview but before saving?

Sorry, we didn't instrument for the elements on the toolbar. How important this kind of information is? Perhaps we can add some tracking in the next release.

This would be great to add to the next release, less so potentially for iOS, but Mobile Web will be working on a new toolbar soon it'd be great to be able to share this data with them to help inform their product decisions.

That should work! So we can extrapolate who dropped off in the Preview and Edit summary screens by looking at the difference between who dropped after preview but before saving?

We know who reach the the preview screen and then who try to save the edit. Taking the difference of them would tell us who drop between preview and save, but users can drop after seeing the preview, or after seeing the edit summary screen -- we can't tell these two apart.

This would be great to add to the next release, less so potentially for iOS, but Mobile Web will be working on a new toolbar soon it'd be great to be able to share this data with them to help inform their product decisions.

Let's talk with @JMinor about how to prioritize this.

JMinor triaged this task as Medium priority.Mar 21 2019, 7:03 PM

Hey @chelsyx the focus on the annual plan metrics looks good. @cmadeo we're going to create a couple additional data tasks for tracking tool usage.

Hey @JMinor, wanted to update you on timing: Chelsy is working on the es.wiki campaign analysis (T215995) and she's also on the hiring panel for the Sr Data Analyst, which has taken up a few hours of her time last week and this week. She's going to touch base with you next week when you're back in the office.

chelsyx moved this task from Next Up to Doing on the Product-Analytics board.Apr 4 2019, 5:07 PM

Done. Please see the results at: https://docs.google.com/presentation/d/1JBs_XyGKdYERpYR0BjMOEjf6N_WINh2pUTJtHcn6L6c/edit?usp=sharing

Key take-aways: Overall, we didn’t see any significant impact of the wikitext editing tools on the annual plan metrics.

  • We didn’t see a big change in total iOS edits:
    • Wikidata description edits increased a lot because we turn on this feature for English Wikipedia again.
    • Wikipedia edits slightly dropped, but it may be a result of seasonality.
  • We didn’t see a big change in active iOS editors, iOS edits revert rate and the edits per editor distribution.
  • For the new iOS editor retention, since it is a long term metric, we don’t expect to see any change relating to the new release yet.
  • We are happy to see that the number of active iOS editors remained high since the Wikidata description release (slide 10).
chelsyx closed this task as Resolved.May 6 2019, 4:05 AM