Page MenuHomePhabricator
Create Task
Maniphest T163107

Flagged revs Special:ValidationStatistics update doesn't work
Open, MediumPublic
Actions

Description

Flagged revision page review statistics (Special:ValidationStatistics) timestamp "The following data was last updated on 17 April 2017 at 08:07. " tells when page is generated. However the data which is shown on the page can be old .

In example Hungarian Wikipedias page reviews statistics says that "The average wait for edits by users that have not logged in to be reviewed is 25 h 59 min; the median is 3 h 38 min. " and based on db average 25.59 has been same from 14.4.2015 and median 3 h 38 from 13.12.2014. The data in percentile table is also years old.

Based on tool labs database case is that in backend stats are updated but for some reason something keeps inserting identical values to database. Bug effects least all Wikipedias with Flagged revs enabled.

Broken function is likely backend/FlaggedRevsStats.php:getEditReviewTimes() as only reviewLag-anon-* and reviewLag-user-* values in flaggedrevs-statistics table in database are broken. Other values are correclty logged to database.

Details

SubjectRepoBranchLines +/-
Cleanup cache update logic in SpecialValidationStatisticsmediawiki/extensions/FlaggedRevsmaster+21 -30
Restore default $wgFlaggedRevsStatsAge (2 hours)operations/mediawiki-configmaster+0 -2
Add some debugging output to FlaggedRevsStatsmediawiki/extensions/FlaggedRevsmaster+9 -0
Enable ValidationStatistics log for FlaggedRevsoperations/mediawiki-configmaster+1 -0
Customize query in gerrit

Related Objects

Event Timeline

Zache created this task.Apr 17 2017, 12:12 PM
Restricted Application added subscribers: TerraCodes, Aklapper. · View Herald TranscriptApr 17 2017, 12:12 PM
Zache added a comment.Apr 17 2017, 12:17 PM

Some timestamps from the DB

reviewLag-anon-median
Quarry: https://quarry.wmflabs.org/query/18063

+-----------+-----------------------+----------------+----------------------------------+--------------+
| wiki      | frs_stat_key          | frs_timestamp  | oldest_timestamp_with_this_value | frs_stat_val |
+-----------+-----------------------+----------------+----------------------------------+--------------+
| alswiki_p | reviewLag-anon-median | 20170417120002 | 20121115230421                   |         1274 |
| ruwiki_p  | reviewLag-anon-median | 20170417100730 | 20161202182402                   |      1174770 |
| dewiki_p  | reviewLag-anon-median | 20170417120040 | 20161021181719                   |         7309 |
| plwiki_p  | reviewLag-anon-median | 20170417100711 | 20140523221756                   |         5689 |
| eowiki_p  | reviewLag-anon-median | 20170417100422 | 20121116063111                   |        13615 |
| huwiki_p  | reviewLag-anon-median | 20170417100658 | 20141213061507                   |        13113 |
| arwiki_p  | reviewLag-anon-median | 20170417120005 | 20131116140010                   |        96130 |
| iawiki_p  | reviewLag-anon-median | 20170417100659 | 20121116074414                   |      1206663 |
| hiwiki_p  | reviewLag-anon-median | 20170417100655 | 20121116072430                   |            0 |
| sqwiki_p  | reviewLag-anon-median | 20170417100738 | 20121116113209                   |        26144 |
| idwiki_p  | reviewLag-anon-median | 20170417100701 | 20140824141611                   |       725916 |
| kawiki_p  | reviewLag-anon-median | 20170417100703 | 20170325140649                   |         4713 |
| fiwiki_p  | reviewLag-anon-median | 20170417100652 | 20130223200302                   |         1115 |
| bnwiki_p  | reviewLag-anon-median | 20170417120008 | 20121116000223                   |            0 |
| trwiki_p  | reviewLag-anon-median | 20170417100743 | 20170408220809                   |        25667 |
| ukwiki_p  | reviewLag-anon-median | 20170417100750 | 20170415180832                   |      1289207 |
| vecwiki_p | reviewLag-anon-median | 20170417100752 | 20160923040721                   |        40167 |
| bewiki_p  | reviewLag-anon-median | 20170417120007 | 20170205140012                   |        54466 |
| enwiki_p  | reviewLag-anon-median | 20170417100059 | 20121116055234                   |            0 |
| ptwiki_p  | reviewLag-anon-median | 20170417100718 | 20131011201949                   |            0 |
| fawiki_p  | reviewLag-anon-median | 20170417100556 | 20140522201549                   |            0 |
| cewiki_p  | reviewLag-anon-median | 20170417120011 | 20160228180014                   |        32295 |
| mkwiki_p  | reviewLag-anon-median | 20170417100705 | 20130513160157                   |         6873 |
+-----------+-----------------------+----------------+----------------------------------+--------------+

reviewLag-anon-average
https://quarry.wmflabs.org/query/18064

+-----------+------------------------+----------------+----------------------------------+--------------+
| wiki      | frs_stat_key           | frs_timestamp  | oldest_timestamp_with_this_value | frs_stat_val |
+-----------+------------------------+----------------+----------------------------------+--------------+
| alswiki_p | reviewLag-anon-average | 20170417120002 | 20121115230421                   |       118624 |
| ruwiki_p  | reviewLag-anon-average | 20170417100730 | 20161202182402                   |      1719487 |
| dewiki_p  | reviewLag-anon-average | 20170417120040 | 20161021181719                   |        28481 |
| plwiki_p  | reviewLag-anon-average | 20170417100711 | 20160714160525                   |        34551 |
| eowiki_p  | reviewLag-anon-average | 20170417100422 | 20121116063111                   |       417771 |
| huwiki_p  | reviewLag-anon-average | 20170417100658 | 20150414181716                   |        93577 |
| arwiki_p  | reviewLag-anon-average | 20170417120005 | 20161012080008                   |       304493 |
| iawiki_p  | reviewLag-anon-average | 20170417100659 | 20121116074414                   |       892960 |
| hiwiki_p  | reviewLag-anon-average | 20170417100655 | 20121116072430                   |            0 |
| sqwiki_p  | reviewLag-anon-average | 20170417100738 | 20131201142208                   |        17419 |
| idwiki_p  | reviewLag-anon-average | 20170417100701 | 20140824141611                   |      3468755 |
| kawiki_p  | reviewLag-anon-average | 20170417100703 | 20170416180716                   |       133599 |
| fiwiki_p  | reviewLag-anon-average | 20170417100652 | 20130223200302                   |         6196 |
| bnwiki_p  | reviewLag-anon-average | 20170417120008 | 20121116000223                   |            0 |
| trwiki_p  | reviewLag-anon-average | 20170417100743 | 20170417100743                   |       392874 |
| ukwiki_p  | reviewLag-anon-average | 20170417100750 | 20170415180832                   |     10547028 |
| vecwiki_p | reviewLag-anon-average | 20170417100752 | 20160923040721                   |       192806 |
| bewiki_p  | reviewLag-anon-average | 20170417120007 | 20170207120010                   |       279930 |
| enwiki_p  | reviewLag-anon-average | 20170417120057 | 20121116055234                   |            0 |
| ptwiki_p  | reviewLag-anon-average | 20170417100718 | 20131011201949                   |            0 |
| fawiki_p  | reviewLag-anon-average | 20170417100556 | 20140522201549                   |            0 |
| cewiki_p  | reviewLag-anon-average | 20170417120011 | 20160228180014                   |      5751210 |
| mkwiki_p  | reviewLag-anon-average | 20170417100705 | 20130513160157                   |       496827 |
+-----------+------------------------+----------------+----------------------------------+--------------+
Zache added a comment.Apr 17 2017, 12:25 PM

Problem is most likely related to backend/FlaggedRevsStats.php: getEditReviewTimes() -function.

Zache renamed this task from Flagged revs page review statistics update doesn't work to Flagged revs Special:ValidationStatistics update doesn't work.Apr 18 2017, 2:28 AM
Zache updated the task description. (Show Details)
Zache mentioned this in T44360: Provide regular cross-wiki reports on flagged revisions status.Apr 18 2017, 2:48 AM
Nemo_bis subscribed.May 3 2017, 9:22 AM

Are those queries you made in quarry the same that FlaggedRevs uses?

The table with the breakdown by namespaces and the table with usernames are correct, aren't they? I can't see on hu.wiki because I don't have enough permissions, but on https://fi.wikipedia.org/wiki/Special:ValidationStatistics the numbers match with what I see in the linked special pages (e.g. 13 unreviewed templates).

Zache added a comment.May 3 2017, 10:07 AM

Database table is same that FlaggedRevs uses but queries are my own and i didn't check the FR code when i made them.

The breakdown by namespaces and the table with usernames should be correct. Also "The average review delay for pages with edits currently pending review" ( pendingLag-average in database) should be ok too.

All other review delay breakdowns are broken. Also Number of users with Editor/autoreviewer rights is broken in some wiki configs but when it works the number is correct. (T163102)

Nemo_bis mentioned this in T163197: Flagged revisions discussion: Current state and future plans.May 19 2017, 9:32 PM
gerritbot subscribed.May 19 2017, 11:07 PM

Change 354608 had a related patch set uploaded (by Nemo bis; owner: Federico Leva):
[operations/mediawiki-config@master] Restore default $wgFlaggedRevsStatsAge (2 hours)

https://gerrit.wikimedia.org/r/354608

gerritbot added a project: Patch-For-Review.May 19 2017, 11:07 PM
Nemo_bis added a comment.May 19 2017, 11:29 PM

This is the query FlaggedRevsStats::getMeanPendingEditTime() and it gives the same results as shown on https://ia.wikipedia.org/wiki/Special:Statisticas_de_validation for instance, as you said.

SELECT AVG( NOW() - UNIX_TIMESTAMP(fp_pending_since) )/3600/24 as avgDaysWait
FROM flaggedpages
JOIN page
ON fp_pending_since IS NOT NULL
AND fp_page_id = page_id
AND page_namespace IN (0, 6, 10);
gerritbot added a comment.May 20 2017, 12:57 AM

Change 354614 had a related patch set uploaded (by Nemo bis; owner: Federico Leva):
[mediawiki/extensions/FlaggedRevs@master] Add some debugging output to FlaggedRevsStats

https://gerrit.wikimedia.org/r/354614

Nemo_bis triaged this task as Medium priority.May 20 2017, 12:58 AM
gerritbot added a comment.May 20 2017, 1:08 AM

Change 354615 had a related patch set uploaded (by Nemo bis; owner: Federico Leva):
[operations/mediawiki-config@master] Enable ValidationStatistics log for FlaggedRevs

https://gerrit.wikimedia.org/r/354615

Nemo_bis removed a project: Patch-For-Review.May 20 2017, 1:09 AM
Liuxinyu970226 subscribed.Feb 14 2018, 7:29 AM
Liuxinyu970226 added a project: Growth-Team.Nov 21 2018, 2:29 PM
Wargo merged a task: T241993: Special:ValidationStatistics doesn't work properly.Jan 6 2020, 4:19 PM
Wargo added a subscriber: Tufor.
Wargo added subscribers: Wargo, PeterBowman.Jan 6 2020, 4:25 PM
TerraCodes unsubscribed.Jan 7 2020, 3:44 AM
Tufor added a comment.Jan 9 2020, 5:17 PM

anyone?

Aklapper added a comment.Jan 9 2020, 9:29 PM

@Tufor: T185664.

Tufor added a comment.Jan 9 2020, 11:06 PM

@Aklapper: so I've read all that discussion and apparently this extension has no assigned owner/maintainer; therefore this bug won't be fixed in the months/years to come, right?

Aklapper added a comment.Jan 10 2020, 6:17 AM

@Tufor: For the start, someone would have to review the patches linked in this task... but finding folks will be hard currently, yes.

gerritbot added a comment.Feb 12 2020, 8:14 PM

Change 354615 abandoned by Jforrester:
Enable ValidationStatistics log for FlaggedRevs

Reason:
Dependency still isn't merged after three years.

https://gerrit.wikimedia.org/r/354615

gerritbot added a comment.Feb 12 2020, 8:14 PM

Change 354614 abandoned by Jforrester:
Add some debugging output to FlaggedRevsStats

Reason:
Abandonware patch, no fix in three years.

https://gerrit.wikimedia.org/r/354614

Nemo_bis added a comment.Feb 12 2020, 8:39 PM

change 354608:

Is this still wanted?

Updating the statistics more often is definitely wanted. I've not repeated the assessment above on how stale the data is. There's certainly still a lot of outdated configuration settings in the FlaggedRevs file.

gerritbot added a comment.Feb 13 2020, 3:13 AM

Change 571853 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/extensions/FlaggedRevs@master] Cleanup cache update logic in SpecialValidationStatistics

https://gerrit.wikimedia.org/r/571853

gerritbot added a project: Patch-For-Review.Feb 13 2020, 3:13 AM
gerritbot added a comment.Feb 13 2020, 7:44 PM

Change 354608 abandoned by Jforrester:
Restore default $wgFlaggedRevsStatsAge (2 hours)

Reason:
Per Aaron.

https://gerrit.wikimedia.org/r/354608

Zache updated the task description. (Show Details)Feb 14 2020, 9:06 AM
Reedy mentioned this in T269196: flaggedrevs_statistics is never purged.Dec 2 2020, 3:21 AM
Wugapodes moved this task from Backlog to Under review on the MediaWiki-extensions-FlaggedRevs board.Mar 16 2021, 9:14 PM
MBinder_WMF added a project: Growth-Team-Filtering.Apr 15 2021, 6:56 PM
Tgr mentioned this in T283148: Special:ValidationStatistics wait time stats are confusing.May 19 2021, 9:37 AM
gerritbot added a comment.Oct 5 2022, 8:24 PM

Change 571853 abandoned by Aaron Schulz:

[mediawiki/extensions/FlaggedRevs@master] Cleanup cache update logic in SpecialValidationStatistics

Reason:

https://gerrit.wikimedia.org/r/571853

Zache mentioned this in T323495: flaggedrevs_statistics is holding millions of rows while FR needs only a handful.Nov 21 2022, 1:50 PM
Pppery removed a project: Patch-For-Review.Oct 29 2023, 6:37 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL