many statistics have fallen to 0 on azwiktionary, ruwikiquote, and ptwikisource
Closed, ResolvedPublic

Description

For some reason, many of the on-wiki statistics (articles, users, edits...) on the Azerbaijani Wiktionary, Russian Wikiquote, and Portuguese Wikisource have inexplicably fallen to 0 sometime in the last 27 hours.

To be specific:

https://az.wiktionary.org/wiki/Special:Statistics shows:

Content pages	0
Pages
(All pages in the wiki, including talk pages, redirects, etc.)	0
Uploaded files	0
Edit statistics
Page edits since Wiktionary was set up	0
Average edits per page	0.00
User statistics
Registered users	0
Active users (list of members)
(Users who have performed an action in the last 30 days)	0
Bots (list of members)	11
Administrators (list of members)	2
Bureaucrats (list of members)	0
Stewards (list of members)	0
Account creators (list of members)	0
Importers (list of members)	0
Transwiki importers (list of members)	0
IP block exemptions (list of members)	0
Oversighters (list of members)	0
Confirmed users (list of members)	0
Structured Discussions bots (list of members)	1
Check users (list of members)	0
Other statistics
Words in all content pages	902,211
Queued mass messages	0
Content pages	0
Pages
(All pages in the wiki, including talk pages, redirects, etc.)	0
Edit statistics
Page edits since Викицитатник was set up	0
Average edits per page	0.00
User statistics
Registered users	0
Active users (list of members)
(Users who have performed an action in the last 30 days)	0
Bots (list of members)	22
Administrators (list of members)	10
Bureaucrats (list of members)	3
Editors (list of members)	52
Reviewers (list of members)	2
Autochecked users (list of members)	0
Account creators (list of members)	0
IP block exemptions (list of members)	3
Oversighters (list of members)	0
Автопатрулируемые (list of members)	66
Confirmed users (list of members)	0
Structured Discussions bots (list of members)	1
Check users (list of members)	0
Other statistics
Words in all content pages	2,702,973
Queued mass messages	0
Content pages	0
Pages
(All pages in the wiki, including talk pages, redirects, etc.)	0
Uploaded files	0
Edit statistics
Page edits since Wikisource was set up	0
Average edits per page	0.00
User statistics
Registered users	0
Active users (list of members)
(Users who have performed an action in the last 30 days)	0
Bots (list of members)	8
Administrators (list of members)	1
Bureaucrats (list of members)	0
Editors (list of members)	34
Reviewers (list of members)	12
Autochecked users (list of members)	26
Stewards (list of members)	0
Account creators (list of members)	0
Importers (list of members)	0
Transwiki importers (list of members)	0
IP block exemptions (list of members)	0
Oversighters (list of members)	0
Confirmed users (list of members)	0
Structured Discussions bots (list of members)	1
Check users (list of members)	0
Other statistics
Words in all content pages	23,726,614
Queued mass messages	0

For reference, the total-page and article counts for these wikis yesterday (c. 2018-02-09 02:00 UTC) were, respectively:

  • wikt:az: 62,198 and 54,876
  • q:ru: 33,489 and 11,877
  • s:pt: 98,335 and 30,353

These appear to be the only Wikimedia content wikis affected in this way.

Anyone know what has caused this and how it should be fixed?

Dcljr created this task.Feb 10 2018, 5:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2018, 5:50 AM
Krinkle triaged this task as Unbreak Now! priority.Feb 10 2018, 6:52 AM
Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptFeb 10 2018, 6:52 AM
Krinkle updated the task description. (Show Details)Feb 10 2018, 6:53 AM
Krinkle added subscribers: aaron, Krinkle.

The only recent commit to files relating to Special:Statistics, SiteStats, showSiteStats, or initSiteStats is:

/cc-ing @aaron.

There is still the correct data in the database:

select * from site_stats;
ss_row_idss_total_viewsss_total_editsss_good_articlesss_total_pagesss_usersss_imagesss_active_users
azwiktionary1036199354876622004743-112
ruwikiquote10302725118793349973368-294
ptwikisource10435841303529834623822-3919

Presumably the incorrect negative values for ss_images cause everything else to be ignored.

Change 409523 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Make SiteStatsInit::doPlaceholderInit() use 1 for ss_row_id

https://gerrit.wikimedia.org/r/409523

There is still the correct data in the database:

select * from site_stats;
ss_row_idss_total_viewsss_total_editsss_good_articlesss_total_pagesss_usersss_imagesss_active_users
azwiktionary1036199354876622004743-112
ruwikiquote10302725118793349973368-294
ptwikisource10435841303529834623822-3919

Presumably the incorrect negative values for ss_images cause everything else to be ignored.

The question is, was this already invalid before the deployment of 3f05f346a8e? Or is it unrelated?

The only effective change from 3f05f346a8e seems to be that, previously, in case of invalid data, we rebuild on-demand even in miser mode (wmf-prod). Whereas now we just insert a zero-filled row and don't recompute anything. However, given you still queried this data after the bug was known, that presumably the "invalid case" wasn't triggered as otherwise there'd be a blank row now, right?

I'm confused.

Change 409523 merged by jenkins-bot:
[mediawiki/core@master] Make SiteStatsInit::doPlaceholderInit() use 1 for ss_row_id

https://gerrit.wikimedia.org/r/409523

I have no idea, I just wanted to see if we actually lost this data. I could go and find out how it ends up not being shown, but hopefully @aaron knows and will enlighten us.

Imarlier moved this task from Inbox to Doing on the Performance-Team board.Feb 12 2018, 8:48 PM
demon removed a subscriber: demon.Feb 12 2018, 9:26 PM

Totally naive suggestion: Could this be fixed by simply recounting images on the affected wikis (so the counts in the database are no longer negative)? Or could that cause other problems?

Change 410425 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@wmf/1.31.0-wmf.20] Make SiteStatsInit::doPlaceholderInit() use 1 for ss_row_id

https://gerrit.wikimedia.org/r/410425

Change 410425 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.20] Make SiteStatsInit::doPlaceholderInit() use 1 for ss_row_id

https://gerrit.wikimedia.org/r/410425

Change 410616 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Salvage site_stats row with negative values in miser mode

https://gerrit.wikimedia.org/r/410616

Mentioned in SAL (#wikimedia-operations) [2018-02-14T23:39:39Z] <AaronSchulz> Running initSiteStats.php on s3 for T186947

Dcljr added a comment.EditedFeb 15 2018, 8:05 AM

FYI: Because s3 contains Wikibooks wikis, which are not part of the monthly article recount that has been happening since 2015 (T68867), those wikis have just seen their first recounting since maybe forever (?). As a result, there have been significant changes in article counts (and total pages, and even edits) on those wikis. See m:Wikimedia_News#February_2018 for (some of) the gory details. (I can post "full" details, like I did for the article recounts of 29 March 2015, later if someone wants them.)

BTW, I believe there may be differences in how initSiteStats counts articles versus how updateArticleCount.php does it, but I'm not sure. Someone may want to look into that. (ISTR that "comma" based counting is not done properly, which would affect enwikibooks and ptwikibooks. Sorry, but I don't know if those are on s3.)

Oh, yeah... enwikibooks and ptwikibooks did just experience large changes in article counts. So they must be on s3.

aaron closed this task as Resolved.Feb 15 2018, 11:01 PM
aaron claimed this task.

Change 410616 merged by jenkins-bot:
[mediawiki/core@master] Salvage site_stats row with negative values in miser mode

https://gerrit.wikimedia.org/r/410616