Page MenuHomePhabricator
Create Task
Maniphest T226661

Create AMC KPIs metrics report (end of FY18-19)
Closed, ResolvedPublic
Actions

Description

Create metrics report on the AMC KPIs status as of the end of FY18-19:

Metric: Mobile web edit rate on target wikis.

  • Target: 10% increase from last year (overall metric).
  • Measured using AMC edits tags done in T212959.
  • To review: eswiki, arwiki, idwiki, thwiki, itwiki, jawiki, fawiki (overall, 100+ edits, 500+ edits cumulative).

Metric: Retention rate for opt-in advanced mobile mode amongst medium and high-volume editors (100+ edits, 500+ edits).

  • Target: At least 60% retention.
  • Measured using the opt-in button - done in T211197 using the PrepUpdate mf_amc_optin. Set to true each time someone opts out.
  • Dashboard with the mf-amc-optin property.

To review: eswiki, arwiki, idwiki, thwiki, itwiki, jawiki, fawiki (overall, 100+ edits, 500+ edits cumulative).

Metric: Moderation actions on mobile web on target wikis.

  • Target: 10% increase from last year - via the logs - not looking at AMC only but overall.
  • Moderation actions are defined in T213461
  • To review: eswiki, arwiki, idwiki, thwiki, itwiki, jawiki, fawiki (overall)
NOTE: AMC still being deployed. Will create a separate task to recalculate these metrics following full rollout and promotion in Q1.

From prior task:
Write and vet a (multi)query to determine the rate of moderation actions (defined in T213461) made using the standard mobile web interface or the AMC interface. For a specific wiki and across all target wikis. This will also serve to vet that the edit tags for these actions are not only appearing correctly in the public log pages (T215597) but also make it to the analytics databases as required for analysis.
A main purpose is to calculate the corresponding annual plan metric (T210660 ).

Signoff criteria

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMNeislerT234559 AMC KPI Report Q2
ResolvedMNeislerT226661 Create AMC KPIs metrics report (end of FY18-19)
 DuplicateMNeislerT220148 Count moderation actions

Event Timeline

MNeisler created this task.Jun 26 2019, 7:02 PM
MNeisler moved this task from Triage to Next Up on the Product-Analytics board.Jun 26 2019, 7:12 PM
MNeisler mentioned this in T226135: Add relevant Web annual plan reports to MediaWiki report page.Jun 26 2019, 8:57 PM
ovasileva triaged this task as High priority.Jun 26 2019, 11:00 PM
ovasileva added a project: Readers-Web-Backlog (Tracking).
ovasileva moved this task from Untriaged to Analyst Consultation on the Readers-Web-Backlog (Tracking) board.
kzimmerman added a subtask: T220148: Count moderation actions.Jun 28 2019, 6:51 PM
MNeisler moved this task from Next Up to Doing on the Product-Analytics board.Jun 28 2019, 8:19 PM
kzimmerman reopened this task as Open.Jul 9 2019, 8:39 PM
kzimmerman closed this task as a duplicate of T220148: Count moderation actions.
kzimmerman merged a task: T220148: Count moderation actions.
kzimmerman added subscribers: • Tbayer, Aklapper.
kzimmerman updated the task description. (Show Details)Jul 9 2019, 8:42 PM
kzimmerman removed subscribers: Aklapper, • Tbayer.
MNeisler added a subscriber: ovasileva.Jul 15 2019, 7:31 PM

@ovasileva: Quick question- For the mobile edit rate and moderation action metrics, would you like to look at the overall rate across all target wikis identified for the project as defined here or just across all of the target wikis where AMC has been deployed to date for this report?

ovasileva added a comment.Jul 16 2019, 12:25 AM
In T226661#5334546, @MNeisler wrote:

@ovasileva: Quick question- For the mobile edit rate and moderation action metrics, would you like to look at the overall rate across all target wikis identified for the project as defined here or just across all of the target wikis where AMC has been deployed to date for this report?

@MNeisler - I think for now we can go with just the wikis where AMC has been deployed

MNeisler updated the task description. (Show Details)Jul 16 2019, 10:59 PM
MNeisler added a comment.Jul 18 2019, 8:22 AM

@ovasileva Here's the current report of the AMC KPIs. Let me know if you have any questions or need any additional info/adjustments for next week's Product meeting.

Report| Repo

Summary of results below:

Data Notes: The data is from the June 2018-June 2019 timeframe and all target wikis where AMC has been deployed( "arwiki", "eswiki", "idwiki", "itwiki", "jawiki", "fawiki", "thwiki"). Noted YOY increases are comparing June 2018 to June 2019 months.

Mobile web edit rate on target wikis (target 10% increase)

  • There was a 24.8% YoY in overall mobile web edits from target wikis.
  • For 100+ and 500+ editor groups, there was a 40% and 37% overall YOY increase respectively on all target wikis.
  • There was a YoY increase in the mobile web edit rate for each target wiki except for Arabic Wikipedia which had a small drop (-7.4%). The highest YoY increase was on Persian Wikipedia (57.8%).
  • I also looked to see what proportion of mobile web edits were tagged with AMC. AMC tags currently just represent a small percent of all mobile web edits (1.4%) on Spanish, Arabic and Indonesian Wikis. This will likely increase following promotion.

mobile_web_edits_overall_daily.png (1×2 px, 224 KB)

Retention rate for opt-in advanced mobile mode

  • There was an 81% overall retention rate across all target wikis where AMC was deployed (time period: March 20, 2019-June 30, 2019).
  • On target wikis, retention rates ranged from 72.6% (Japanese Wikipedia) to 90.8% (Persian Wikipedia).
  • There was a 70.8% overall retention rate limited to users with 100+ and 61.4% rate limited to users with 500+ edits (time period: March 20, 2019-June 30, 2019)

amc_retention_daily.png (1×2 px, 209 KB)

amc_retention_prop_bywiki.png (1×2 px, 64 KB)

Moderation actions on mobile web on target wikis (overall)
Note: It looks not all moderation actions were correctly tagged with mobile web edit prior to Feb/March 2019 (I think it may be related to this note/task. As a result, I am currently unable to show a YoY increase but have documented trends from March 2019 to now. I'll look into this further and see if there might be another way to get earlier data.

  • There is a sudden spike in actions around March 2019 but that is likely due to the timing of when mobile web edits tags were added to all moderation actions. There are no significant changes in the rate of each moderation action type use from March 2019 to June 2019.
  • Both the thank and the block actions are the most commonly used moderation actions on mobile web in the target countries. There were no records of delete, protect and unblock actions with mobile web edit tags (will need to investigate further).
  • Italian Wikipedia had the highest number of moderation actions on mobile web the past year (5,724) while Thai Wikipedia had the lowest (176).
MNeisler updated the task description. (Show Details)Jul 23 2019, 12:15 AM
ovasileva updated the task description. (Show Details)Jul 23 2019, 2:24 PM
MNeisler added a comment.Jul 23 2019, 7:56 PM

I made some small edits to indicate the date when the mobile web and amc tags were added to action in the log table and did some final report cleanup. There are no changes to the numbers previously reported. Please see final report. Let me know if you have any questions.

Per our discussions, I'll rerun this report a couple months after the full deployment and promotion of AMC.

MNeisler closed this task as Resolved.Jul 26 2019, 4:20 PM
This comment was removed by MNeisler.
MNeisler reopened this task as Open.Jul 26 2019, 4:33 PM
ovasileva closed this task as Resolved.Aug 5 2019, 9:24 PM

Report is complete and published, resolving this. Thanks @MNeisler!

ovasileva added a parent task: T234559: AMC KPI Report Q2 .Oct 3 2019, 7:21 PM
Restricted Application added subscribers: Huji, Masumrezarock100. · View Herald TranscriptJan 30 2022, 9:44 AM
• Kokkiknamtan reopened this task as In Progress.Jan 30 2022, 9:46 AM
• Kokkiknamtan set Due Date to Jan 29 2022, 5:00 PM.
• Kokkiknamtan reopened subtask T220148: Count moderation actions as Open.Jan 30 2022, 10:13 AM
• Kokkiknamtan added a subscriber: • Kokkiknamtan.Jan 30 2022, 10:31 AM
In T226661#5394651, @ovasileva wrote:

Report is complete and published, resolving this. Thanks @MNeisler!

Reedy closed this task as Resolved.Jan 30 2022, 12:30 PM
Reedy merged a task: T220148: Count moderation actions.
Reedy added subscribers: • Tbayer, Aklapper.
Aklapper removed Due Date.Jan 30 2022, 9:55 PM
Aklapper removed a subscriber: • Kokkiknamtan.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL