Page MenuHomePhabricator

Discuss Wikistats integration for ORES
Open, LowestPublic

Description

The Wikistats 2.x platform would be a good place to surface high-level ORES rollups, e.g. total number of "damaging" edits.

This task is complete when we have completed discussion and either abandon the idea or create tasks for implementation.

Event Timeline

awight created this task.Jan 8 2018, 9:31 PM
Restricted Application added a project: Analytics. · View Herald TranscriptJan 8 2018, 9:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Nuria moved this task from Incoming to Radar on the Analytics board.Jan 11 2018, 5:52 PM

totally, put a meeting on our calendar or let's chat here.

ping @awight still want to chat about this?

awight added a subscriber: Halfak.Apr 30 2018, 5:21 PM

@Milimetric Sorry to miss the earlier ping. Some top-level metrics we can offer, (@Halfak please chime in here):

  • Total number of predicted damaging edits per time
  • Proportion of predicted damaging edits per time
  • Same as above, but for good-faith
  • Predicted average article quality over time
  • Derivative of predicted average article quality (is our wikis learning?)
  • Proportion of drafts in each predicted class: OK, spam, vandalism, attack

These stats will only be available for wikis supported by various ORES models, see this table for an exact mapping: https://tools.wmflabs.org/ores-support-checklist/

Halfak added a comment.May 1 2018, 2:52 PM

One that I think would be interesting for Wikistats is a count of the number of non-redirect main namespace articles that fall into a set of predicted categories. We can generate this historically and then monthly from there forward. We already have a process for generating the raw data (see https://figshare.com/articles/Monthly_Wikipedia_article_quality_predictions/3859800) and there's interest in using this as a coverage/gap metric.

Milimetric added a comment.EditedMay 10 2018, 8:52 PM

@awight. Those look great and in my opinion they fit right in for the most part. We organize metrics into three groups:

content: the content itself: size, growth, etc.
contributing: focused on the editors: top editors, counts by activity level, etc.
reading: pageviews, uniques, etc.

So, by that organization here's how I see the metrics you two mentioned:

  • Total number of predicted damaging edits per time

This can go into contributing, I would call it "Predicted Edits" or something like that, and include "damaging", "good-faith", and anything else as breakdowns of the total number, and there are multiple reasons for that. In future versions of the UI you'll be able to do more to see just one versus the other though, so we should talk if this seems to make no sense.

  • Proportion of predicted damaging edits per time

This would be more useful than the above probably, but we want to build an advanced interface where you can compute ratios like this on the fly, so for now we just want the basic numbers like the total.

  • [Total number of predicted] good-faith [per time]

I would bundle this with the above into Predicted Edits

  • Predicted average article quality over time

I'd put this in Content and call it Predicted Article Quality

  • Derivative of predicted average article quality (is our wikis learning?)

I'd leave this for the advanced interface again, then people can take the derivative of whatever they like

  • Proportion of drafts in each predicted class: OK, spam, vandalism, attack

Also Content, maybe "Predicted Draft Class"? The different classes would be breakdowns of the total which would be all draft articles

One that I think would be interesting for Wikistats is a count of the number of non-redirect main namespace articles that fall into a set of predicted categories. We can generate this historically and then monthly from there forward. We already have a process for generating the raw data (see https://figshare.com/articles/Monthly_Wikipedia_article_quality_predictions/3859800) and there's interest in using this as a coverage/gap metric.

I'd put this in Content as well, and call it something like Category Prediction Accuracy

Ok, enough fun stuff. So to put this all together, we need to:

  • gather the data into a Hive table or some easy place from where we can ingest it into Druid
  • we need to do the above with some process that's easy to monitor, restart, etc., probably an Oozie job
  • define a Druid schema and ingest the data
  • build queries and endpoints in AQS to get at these metrics (https://github.com/wikimedia/analytics-aqs)

None of these things are hard, and if you want to do them we can definitely help. But if we need to do them, then the work has to get on the annual plan and run the gauntlet: https://phabricator.wikimedia.org/tag/analytics

Other thoughts and responses:

These stats will only be available for wikis supported by various ORES models, see this table for an exact mapping: https://tools.wmflabs.org/ores-support-checklist/

This is fine, we can build UI that will help users understand.

Also, just for the record, I think this would be awesome awesome awesome. But we don't get to drop other stuff just because awesomer stuff comes along :)

238482n375 set Security to Software security bug.Jun 15 2018, 8:06 AM
238482n375 added a project: Security.
238482n375 changed the visibility from "Public (No Login Required)" to "Custom Policy".
238482n375 added a subscriber: 238482n375.
This comment was removed by akosiaris.
akosiaris changed the visibility from "Custom Policy" to "Public (No Login Required)".
akosiaris removed a subscriber: 238482n375.
akosiaris added a subscriber: akosiaris.
Restricted Application added a project: Security. · View Herald TranscriptJun 15 2018, 11:30 AM
Harej triaged this task as Lowest priority.Apr 3 2019, 5:11 AM