Page MenuHomePhabricator

Baseline: Size of content moderation backlog - FlaggedRevs
Closed, ResolvedPublic

Description

Research question: Does Automoderator reduce the workload of human patrollers in countering vandalism?

Evaluation method: Does the volume of various content moderation backlogs reduce?

Baseline
We will measure and compare this data directly pre- and post- deployment. We need to carry out some qualitative investigations first to define which backlogs we are interested in evaluating.


Question: TBD

This analysis should be done individually for the top 10 Wikipedia projects by editing activity: en, es, ja, de, fr, ru, zh, it, pt, fa, plus our pilot id.wiki


Event Timeline

Samwalton9-WMF renamed this task from What is the size of content moderation backlogs? to Baseline: What is the size of content moderation backlogs?.
Samwalton9-WMF changed the task status from Open to Stalled.

@Samwalton9-WMF As we discussed, I have started working on flagged revs and recent changes patrolling. I have results for flagged revisions for now.

I suggest we create a separate task for recent changes. Combining recent changes with patrol logs has been tricky as documentation for log params is unclear. It will need more time.

Per discussion with @Samwalton9-WMF, ruwiki has a large backlog, I will be re-doing the analysis after talking to Amir.

KCVelaga_WMF renamed this task from Baseline: What is the size of content moderation backlogs? to Baseline: Size of content moderation backlog - FlaggedRevs.Jan 9 2024, 11:28 AM
KCVelaga_WMF moved this task from Triage to Current Quarter on the Product-Analytics board.

@Samwalton9-WMF

flaggedrevisions yet to be reviewed
While the following stats give us a good initial insight into the state of flagged revisions, we shouldn't consider these for baseline reference (for revisions yet to be reviewed). These can be very different depending on when the query was run. We can talk about some cadence to regularly calculate these stats and observe the trend over a longer period of time. For example, the query can be run every hour or day, and then the average over a few weeks/months can be used for baseline. However, that will be a separate task to setup a regular processing job.

WikipediaRevisions to be ReviewedTime Elapsed Since Revision (median)
ruwiki3722661 year 9 months 2 days 22 hours 21 minutes 39 seconds
dewiki69371 week 2 days 5 hours 4 minutes 51 seconds
idwiki10113 years 7 months 3 days 11 hours 41 minutes 52 seconds
enwiki41 hour 22 minutes 41 seconds

at the time of the query: 2024-02-15 11:26:24


flaggedrevisions that have been reviewed (2023)

WikipediaMedian Time to to be Reviewed (minutes)
dewiki236.92
enwiki13.18
idwiki125.785
ruwiki274.62
wiki_dbAverage Monthly Unique Reviewers
dewiki2472
enwiki231
idwiki22
ruwiki752
wiki_dbMedian Number of Reviews by Each Unique Reviewer
dewiki7
enwiki3
idwiki5
ruwiki33

Analysis notebook

Average Daily Number of Reviews and Reviewers Flagged Revs (2023)

wiki_db# Reviewers# Reviews
dewiki3331137
enwiki2460
idwiki26
ruwiki171959