Page MenuHomePhabricator

Create AMC KPIs metrics report (end of FY18-19)
Closed, ResolvedPublic

Description

Create metrics report on the AMC KPIs status as of the end of FY18-19:

Metric: Mobile web edit rate on target wikis.

  • Target: 10% increase from last year (overall metric).
  • Measured using AMC edits tags done in T212959.
  • To review: eswiki, arwiki, idwiki, thwiki, itwiki, jawiki, fawiki (overall, 100+ edits, 500+ edits cumulative).

Metric: Retention rate for opt-in advanced mobile mode amongst medium and high-volume editors (100+ edits, 500+ edits).

  • Target: At least 60% retention.
  • Measured using the opt-in button - done in T211197 using the PrepUpdate mf_amc_optin. Set to true each time someone opts out.
  • Dashboard with the mf-amc-optin property.

To review: eswiki, arwiki, idwiki, thwiki, itwiki, jawiki, fawiki (overall, 100+ edits, 500+ edits cumulative).

Metric: Moderation actions on mobile web on target wikis.

  • Target: 10% increase from last year - via the logs - not looking at AMC only but overall.
  • Moderation actions are defined in T213461
  • To review: eswiki, arwiki, idwiki, thwiki, itwiki, jawiki, fawiki (overall)
NOTE: AMC still being deployed. Will create a separate task to recalculate these metrics following full rollout and promotion in Q1.

From prior task:
Write and vet a (multi)query to determine the rate of moderation actions (defined in T213461) made using the standard mobile web interface or the AMC interface. For a specific wiki and across all target wikis. This will also serve to vet that the edit tags for these actions are not only appearing correctly in the public log pages (T215597) but also make it to the analytics databases as required for analysis.
A main purpose is to calculate the corresponding annual plan metric (T210660 ).

Signoff criteria

Event Timeline

@ovasileva: Quick question- For the mobile edit rate and moderation action metrics, would you like to look at the overall rate across all target wikis identified for the project as defined here or just across all of the target wikis where AMC has been deployed to date for this report?

@ovasileva: Quick question- For the mobile edit rate and moderation action metrics, would you like to look at the overall rate across all target wikis identified for the project as defined here or just across all of the target wikis where AMC has been deployed to date for this report?

@MNeisler - I think for now we can go with just the wikis where AMC has been deployed

@ovasileva Here's the current report of the AMC KPIs. Let me know if you have any questions or need any additional info/adjustments for next week's Product meeting.

Report| Repo

Summary of results below:

Data Notes: The data is from the June 2018-June 2019 timeframe and all target wikis where AMC has been deployed( "arwiki", "eswiki", "idwiki", "itwiki", "jawiki", "fawiki", "thwiki"). Noted YOY increases are comparing June 2018 to June 2019 months.

Mobile web edit rate on target wikis (target 10% increase)

  • There was a 24.8% YoY in overall mobile web edits from target wikis.
  • For 100+ and 500+ editor groups, there was a 40% and 37% overall YOY increase respectively on all target wikis.
  • There was a YoY increase in the mobile web edit rate for each target wiki except for Arabic Wikipedia which had a small drop (-7.4%). The highest YoY increase was on Persian Wikipedia (57.8%).
  • I also looked to see what proportion of mobile web edits were tagged with AMC. AMC tags currently just represent a small percent of all mobile web edits (1.4%) on Spanish, Arabic and Indonesian Wikis. This will likely increase following promotion.

Retention rate for opt-in advanced mobile mode

  • There was an 81% overall retention rate across all target wikis where AMC was deployed (time period: March 20, 2019-June 30, 2019).
  • On target wikis, retention rates ranged from 72.6% (Japanese Wikipedia) to 90.8% (Persian Wikipedia).
  • There was a 70.8% overall retention rate limited to users with 100+ and 61.4% rate limited to users with 500+ edits (time period: March 20, 2019-June 30, 2019)


Moderation actions on mobile web on target wikis (overall)
Note: It looks not all moderation actions were correctly tagged with mobile web edit prior to Feb/March 2019 (I think it may be related to this note/task. As a result, I am currently unable to show a YoY increase but have documented trends from March 2019 to now. I'll look into this further and see if there might be another way to get earlier data.

  • There is a sudden spike in actions around March 2019 but that is likely due to the timing of when mobile web edits tags were added to all moderation actions. There are no significant changes in the rate of each moderation action type use from March 2019 to June 2019.
  • Both the thank and the block actions are the most commonly used moderation actions on mobile web in the target countries. There were no records of delete, protect and unblock actions with mobile web edit tags (will need to investigate further).
  • Italian Wikipedia had the highest number of moderation actions on mobile web the past year (5,724) while Thai Wikipedia had the lowest (176).

I made some small edits to indicate the date when the mobile web and amc tags were added to action in the log table and did some final report cleanup. There are no changes to the numbers previously reported. Please see final report. Let me know if you have any questions.

Per our discussions, I'll rerun this report a couple months after the full deployment and promotion of AMC.

This comment was removed by MNeisler.

Report is complete and published, resolving this. Thanks @MNeisler!