Page MenuHomePhabricator
Create Task
Maniphest T234559

AMC KPI Report Q2
Closed, ResolvedPublic
Actions

Assigned To
MNeisler
Authored By
ovasileva
Oct 3 2019, 7:16 PM
Tags
Referenced Files
F31253626: mobile_web_special_page_requests_overall.png
Nov 25 2019, 2:53 PM
F31253655: mobile_web_talk_page_edits_amc_prop.png
Nov 25 2019, 2:53 PM
F30947809: amc_edits_prop_overall_stacked_chart.png
Nov 1 2019, 7:19 PM
F30946926: moderation_counts_enwiki_daily.png
Nov 1 2019, 7:14 PM
F30946782: moderation_counts_daily_overall.png
Nov 1 2019, 7:14 PM
Subscribers
Aklapper
Jdlrobson
kzimmerman
Masumrezarock100
MNeisler
olga
ovasileva

Description

Acceptance Criteria

KPIs:

For the following, review all wikis together, separate by top 10 wikis

Metric: Mobile web edit rate on all wikis.

  • Target: 10% increase from last year (overall metric).
  • Measured using AMC edits tags done in T212959.
  • To review: (overall, 100+ edits, 500+ edits cumulative)

Metric: Retention rate for opt-in advanced mobile mode amongst medium and high-volume editors (100+ edits, 500+ edits).

  • Target: At least 60% retention.
  • Measured using the opt-in button - done in T211197 using the PrepUpdate mf_amc_optin. Set to true each time someone opts out.
  • Dashboard with the mf-amc-optin property.

To review: (overall, 100+ edits, 500+ edits cumulative).

Metric: Moderation actions on mobile web on target wikis.

  • Target: 10% increase from last year - via the logs - not looking at AMC only but overall.
  • Moderation actions are defined in T213461
  • To review: (overall, 100+ edits, 500+ edits cumulative)
Other Metrics
  • Rates of access to special pages overall i.e. pageviews, repeating the query from T198218, relying on new X-Analytics tag for AMC, analogous to the mf-m key or as a new value for it: T212961
  • Rates of access and edits to article talk pages- relying on AMC X-Analytics tag and AMC edit tag

Signoff criteria

NOTE: short-form report w/ highlights by October 24, full report by Nov 1

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMNeislerT234559 AMC KPI Report Q2
ResolvedMNeislerT226661 Create AMC KPIs metrics report (end of FY18-19)
 DuplicateMNeislerT220148 Count moderation actions

Event Timeline

ovasileva created this task.Oct 3 2019, 7:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 3 2019, 7:16 PM
ovasileva triaged this task as Medium priority.Oct 3 2019, 7:16 PM
ovasileva added a project: Readers-Web-Backlog (Tracking).
ovasileva moved this task from Untriaged to Analyst Consultation on the Readers-Web-Backlog (Tracking) board.
ovasileva added a subtask: T226661: Create AMC KPIs metrics report (end of FY18-19).Oct 3 2019, 7:21 PM
ovasileva raised the priority of this task from Medium to High.Oct 15 2019, 1:12 PM
Restricted Application added a subscriber: Masumrezarock100. · View Herald TranscriptOct 15 2019, 1:12 PM
ovasileva updated the task description. (Show Details)Oct 15 2019, 1:13 PM
ovasileva updated the task description. (Show Details)Oct 15 2019, 1:16 PM
ovasileva added a subscriber: MNeisler.
MNeisler claimed this task.Oct 15 2019, 7:32 PM
MNeisler added a project: Product-Analytics.
MNeisler moved this task from Triage to Doing on the Product-Analytics board.
mmodell edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Oct 16 2019, 6:31 PM
mmodell moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Oct 16 2019, 6:32 PM
MNeisler added a comment.Oct 18 2019, 8:50 PM

@ovasileva - A couple questions:

  • Can you confirm the date that AMC was released to all Wikis?
  • In the previous report, we looked at the overall increase in all mobile web edits. Now that AMC has deployed to all wikis for a little while, would it be beneficial to report on the proportion of mobile web edits that are tagged with the AMC edit tag? or both?

I'll shoot to get you updates on the key metrics numbers by early next week (I heard some of these numbers might be needed for the Tuning Session deck).

Masumrezarock100 added a comment.Oct 18 2019, 10:22 PM

@MNeisler AMC was deployed to all Wikimedia wikis on October 10th, 2019. (per T233612)

ovasileva added a comment.Oct 21 2019, 8:43 AM
In T234559#5588013, @MNeisler wrote:

@ovasileva - A couple questions:

  • Can you confirm the date that AMC was released to all Wikis?
  • In the previous report, we looked at the overall increase in all mobile web edits. Now that AMC has deployed to all wikis for a little while, would it be beneficial to report on the proportion of mobile web edits that are tagged with the AMC edit tag? or both?

I'll shoot to get you updates on the key metrics numbers by early next week (I heard some of these numbers might be needed for the Tuning Session deck).

@MNeisler - AMC was deployed to all Wikipedias August 7th (to all non-Wikipedia projects October 10th). And yes :) I think reporting on the proportion of mobile web edits tagged with AMC makes sense rather than looking at overall edit increases.

MNeisler added a comment.EditedOct 23 2019, 4:47 AM

Here's a quick high-level summary of the AMC metric numbers gathered so far. I'll post a moderation actions update soon and a full report with more details by early next week.
I reviewed data starting at the date of AMC deployment (varies based on wiki [1]) through September 30, 2019.

Mobile web edit rate
Proportion of all logged in mobile web edits tagged with the new AMC edit tag.

WikisAll logged in editors100+ Edits500+ Edits
All Wikipedias17.3%18.7%26.3%
English Wiki16.2%18.8%24.9%
TargetWikis (arwiki, eswiki, idwiki)10.0%9.74%15.25%
TargetWikis (itwiki, jawiki, fawiki, thwiki)10.8%10.1%16.7%

amc_edits_prop_overall_stacked_chart.png (1×2 px, 89 KB)

Retention rate (Opt-in Rate)

Proportion of opt-ins for advanced mobile mode

WikisAll logged in editors100+ Edits500+ Edits
All Wikipedias88.3%94.7%87.3%
English Wiki90.8%95.5%89.4%
TargetWikis (arwiki, eswiki, idwiki)82.4%88.3%76.1%
TargetWikis (itwiki, jawiki, fawiki, thwiki)81.7%88.3%74.7%

[1] Deployment dates:
All Wikipedias: August 7, 2019
English Wiki: August 7, 2019
Arwiki, idwiki and eswiki target wikis: March 20, 2019
Itwiki, jawiki, fawiki, thwiki target wikis: June 17, 2019
Non-Wikipedia Projects: October 10, 2019 *Incomplete data for October so did not review for this current report

MNeisler added a comment.EditedNov 1 2019, 7:14 PM

Here is a link to the current draft report, which includes additional per wiki and editor experience breakdowns for the primary KPIs (mobile web edit rate, retention rate, and mobile web moderation actions).

Summary of the changes in moderation actions on mobile web following AMC deployments below:

There has been a steady increase in moderation actions across wikis the past quarter. From Q4 (April to June 2019) to Q1 (July - September 2019), the total number of moderation actions on mobile web increased by 47%. The month over month increase from August to September 2019 was 30% across all wikis.

moderation_counts_daily_overall.png (1×2 px, 138 KB)

On English Wikipedia, there was a significant increase in moderation action on mobile web following the deployment of AMC to all Wikipedia. Comparing the month before deployment (July 2019) to the month after (August 2019), the number of moderation actions on mobile web increased by 46.9% from July to August 2019. There was a 17% month over month increase from August to September 2019.

moderation_counts_enwiki_daily.png (1×2 px, 95 KB)

In regards to type of moderation action, there has been an overall increase in all types of moderation actions on mobile web following the AMC deployment. The thank action is the most commonly used moderation action on mobile web. However, the usage of the block and approve actions have seen significant increases since AMC was deployed on all wikis.

Remaining actions on my to do list until before it's ready to post:

  • Rates of access to special pages overall i.e. pageviews relying on new X-Analytics tag for AMC.
  • Rates of access and edits to article talk pages- relying on AMC X-Analytics tag and AMC edit tag
  • Final review and an overall summary of key observations/trends identified from analysis
Jdlrobson added a subscriber: Jdlrobson.Nov 5 2019, 12:51 AM

Are these new actions or are they actions that we have cannibalised from the desktop site? Are we able to compare desktop for the same period? Regardless these seem like impressive numbers!

MNeisler added a comment.Nov 12 2019, 5:34 PM
In T234559#5634192, @Jdlrobson wrote:

Are these new actions or are they actions that we have cannibalised from the desktop site? Are we able to compare desktop for the same period? Regardless these seem like impressive numbers!

These are any moderation actions that were tagged as mobile web edits or both mobile web and AMC edits so not limited to just new actions. We unfortunately can't easily compare numbers to the desktop because the available tag options do not currently include a tag indicating if an edit or action was made from a desktop device. You could assume that the absence of mobile tag means it's from desktop but there are a number of exceptions (such as revisions made using APIs)

FYI - I'm wrapping up the final report today (with the results for the final two metrics talk page and special page access rates) and will link here as soon as it's ready.

MNeisler added a comment.Nov 25 2019, 2:53 PM

Here is a link to the final AMC metrics report.

Quick summary of results found for the remaining metrics below:

  • Rate of access to special pages : Requests to special pages has remained relatively flat (there was only a 3% month over month change between September 2019 and October 2019) and views from mobile web in AMC mode represent about only 2% of all mobile web views to these pages. However, there has been a slow increase in the proportion of mobile web requests to special pages coming from AMC mode since it was deployed on all Wikipedias on August 7, 2018. The proportion of AMC requests has been increased from September to October 2019 by 15.9%. The Mobile Diff, Watchlist, Contributions, and Search are the most viewed special pages from users in AMC mode on all Wikipedias.

mobile_web_special_page_requests_overall.png (1×2 px, 74 KB)

  • Rate of access and edits to talk pages :
    • There was a 10.6% year over year increase in the number of views to article talk pages from mobile web in October 2019 across all Wikipedias. About 42% of these views came from logged-in users in AMC mode.
    • The rate of edits to talk pages have also continued to increase following the July and August deployment dates. While other variables such as changes in the number of active editors may contribute partially to these increases, the proportion of talk page edits made by users in AMC mode has been increasing each month following deployment. AMC edits accounted for 42.1% of all talk page edits made by logged-in users on mobile web in October 2019.

mobile_web_talk_page_edits_amc_prop.png (1×2 px, 117 KB)

@ovasileva - Let me know if you have any questions or comments.

MNeisler moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.Nov 25 2019, 2:53 PM
MNeisler mentioned this in T239672: Many special pages missing from pageview_hourly dataset starting on July 23, 2019.Dec 3 2019, 12:17 AM
MNeisler added a subscriber: olga.Dec 20 2019, 11:26 PM

@olga Any questions or follow-up actions needed here?

kzimmerman closed this task as Resolved.Jan 6 2020, 5:52 PM
kzimmerman added a subscriber: kzimmerman.

@ovasileva marking this as resolved, but let us know if there are outstanding questions we should consider. can you make sure the necessary info is published on Wiki at this point?

• Kokkiknamtan reopened subtask T226661: Create AMC KPIs metrics report (end of FY18-19) as In Progress.Jan 30 2022, 9:46 AM
Reedy closed subtask T226661: Create AMC KPIs metrics report (end of FY18-19) as Resolved.Jan 30 2022, 12:30 PM
Samwalton9 mentioned this in T305633: [REQUEST] Moderator Tools team support for content moderation on mobile web.Apr 7 2022, 1:43 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL