Page MenuHomePhabricator

Analyze effect of huwiki FlaggedRevs configuration change on problematic edits and new user retention
Open, Needs TriagePublic

Description

On 2018-04-09 FlaggedRevs on Hungarian Wikipedia was reconfigured: before, articles were displayed to readers in their last reviewed version, afterwards articles were always shown in their latest version and FlaggedRevs was only used to coordinate patrolling activity. Now that the test period for the change is over, we need to evaluate the effect and decide whether to keep the change.

The main questions to answer:

  • did the ratio of problematic edits increase after the change?
  • was new editor retention affected?

The secondary question would be to quantify how often readers saw bad changes due to the new configuration, and how often they didn't see new changes due to the old configuration.

Event Timeline

Tgr created this task.Nov 11 2018, 4:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 11 2018, 4:10 AM
Zache added a subscriber: Zache.Nov 11 2018, 11:58 AM

@Tgr: Who to analyze this?

Tgr added a comment.Nov 19 2018, 8:26 AM

@Tgr: Who to analyze this?

The huwiki community (help is of course welcome if anyone is interested :).

Tgr added a comment.Nov 19 2018, 8:42 AM

Registrations:

new registrations, dailyrunning average, 45 daysdiff in running average

(with apologies to everyone who actually knows how to do statistics)

Zache added a comment.Nov 19 2018, 8:27 PM

The 12 month running median is very stable for huwiki. There slight increase in monthly >5 trend after configuration change but in other ways it is almost flat.

Source

Samat added a subscriber: Samat.Nov 24 2018, 11:17 PM
Tgr added a subscriber: Halfak.May 20 2019, 11:33 AM

@Halfak said he has some ideas for this (thanks!)

Tgr added a comment.May 20 2019, 11:34 AM

One thing I would like to do here is an analysis of the number of vandal edits (as estimated by ORES), but I'd like to have more confidence in ORES prediction quality first.

Zache added a comment.May 20 2019, 1:17 PM

I would like to understand what i am actually looking here: (ie in one year huwiki's "active anonymous editors" editing level raised to same level than in fiwiki. Before that they were editing with accounts and now they are making edits without logging in?

https://stats.wikimedia.org/v2/#/hu.wikipedia.org/contributing/active-editors/normal|line|all|editor_type~anonymous*group-bot*name-bot*user|monthly

Tgr added a comment.May 21 2019, 9:04 PM

I'd expect a drop in active logged-in editors in that case, so this does seems like a real increase in contributors. OTOH if you look at editors (https://stats.wikimedia.org/v2/#/hu.wikipedia.org/contributing/editors/normal|line|all|editor_type~anonymous*user|monthly|editors - "active editor" means 5+ edits last month in the wikistats vocabulary, "editor" means 1+ edits last month), they have started growing significantly sooner (although there seems to be an uptick when flagrev was disabled? hard to tell by the naked eye), so there seems to be something else going on there and it's not obvious to what extent is the flagrev switch the cause.

That said I would be mainly interested in changes to the number of new users (more anon edits is not trivially a good thing, although ORES might help in better evaluating it), which shows no obvious change, but new registrations tend to be dominated by users who never edit (and probably never meant to edit in the first place) and that might drown out the signal so we probably need some filtering there to get a meaningful graph.

Zache added a comment.EditedMay 24 2019, 7:15 AM

I updated the flagged revs reviews stats graphs and fiwiki as reference.

Note for fiwiki bot reviews arent filttered out and ~10% of manual reviews of 2018-> are actually made by SeulojaBot (based on ORES and other rules)

Tgr added a comment.Thu, Jun 20, 7:50 PM

(Note to self because someone asked and I had to search for it: the original enabling was in 2009 June: T17568.)

Tgr added a comment.EditedThu, Jun 20, 7:52 PM

Users with 5-24 edits are up too, that's encouraging (unless that stat somehow includes anons): https://stats.wikimedia.org/v2/#/hu.wikipedia.org/contributing/editors/normal|line|all|activity_level~5..24-edits|monthly

Tgr added a comment.Mon, Jul 1, 8:41 AM

I checked the registration year of users with 5/25+ edits in the last 30 days to get an estimate of how much these new editors are converted into long-term productive editors (5+, 25+) but didn't see any clear trend. (The numbers are too small and noisy for a graph to be useful.)

Samat added a comment.Mon, Jul 15, 9:57 PM

I started to prepare a page with a statistical analysis here (in Hungarian only, yet, but mainly with easy to understand figures):
https://hu.wikipedia.org/wiki/Wikip%C3%A9dia:Jel%C3%B6lt_lapv%C3%A1ltozatok/Statisztik%C3%A1k
I will fill up with the main conclusions in the following days (most of them are obvious from the graphs anyway).