Special:AllPages disabled due to performance issues
Closed, ResolvedPublic

Description

After Special:AllPages causing an outage earlier today, it has been temporarily disabled pending investigation.

This ticket serves as the public tracking for the issue

Reedy created this task.Mar 20 2017, 3:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2017, 3:17 PM
Paladox triaged this task as "High" priority.Mar 20 2017, 3:29 PM
Paladox added a subscriber: Paladox.

Should be high priority as it affects prod wiki's.

Special:Allpages still displays MediaWiki:Allpages-summary at the top so wikis can create the message to explain it like I did in https://en.wikipedia.org/wiki/MediaWiki:Allpages-summary and mentioned in https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Special:AllPages_disabled. I guess a temporary MediaWiki default for MediaWiki:Allpages-summary might not be displayed before the issue is fixed.

revi added a subscriber: revi.Mar 20 2017, 3:43 PM
hashar added a subscriber: hashar.Mar 20 2017, 3:59 PM

Mentioned in SAL (#wikimedia-operations) [2017-03-20T15:59:51Z] <hashar> Special:AllPages being blank has a public task: https://phabricator.wikimedia.org/T160916

KTC added a subscriber: KTC.Mar 20 2017, 5:05 PM

Change 343681 had a related patch set uploaded (by Reedy):
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343681

I think it can safely re-enabled on significantly smaller wikis even prior to any investigation. Or better, IMHO, it can be disabled just on a limited set of bigger wikis.

Reedy added a comment.Mar 20 2017, 6:01 PM

I think it can safely re-enabled on significantly smaller wikis even prior to any investigation. Or better, IMHO, it can be disabled just on a limited set of bigger wikis.

Hence the patch to disable the offending feature in miser mode :)

Samuele2002 removed a subscriber: MZMcBride.EditedMar 20 2017, 7:01 PM

Sorry. They are so mobile and when I added has removed everything else. It already happened once when they are mobile. someone tell me why?

Change 343711 had a related patch set uploaded (by Reedy):
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343711

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

We had a user hitting enwiki enmass screenscraping which caused issues.

Reedy added a comment.Mar 20 2017, 7:44 PM

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

We had a user hitting enwiki enmass screenscraping which caused issues.

Well, that and there's potentially some issues in MariaDB's query planner that we may be experiencing post upgrades too

Change 343681 merged by jenkins-bot:
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343681

Change 343711 merged by jenkins-bot:
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343711

Mentioned in SAL (#wikimedia-operations) [2017-03-20T20:12:21Z] <reedy@tin> Synchronized php-1.29.0-wmf.16/includes/specials/SpecialAllPages.php: Re-enable Special:AllPages, disable redirect filter if MiserMode T160916 (duration: 00m 42s)

Mentioned in SAL (#wikimedia-operations) [2017-03-20T20:14:12Z] <reedy@tin> Synchronized php-1.29.0-wmf.16/includes/api/ApiQueryAllPages.php: Limit query=allpages filterredir if MiserMode T160916 (duration: 00m 42s)

there's potentially some issues in MariaDB's query planner

While that is probably true, I believe that the root cause was a not perfect query, in some very specific cases, in some very large wikis, having bad performance- something that only was surfaced when someone tried to scrap all pages with very high concurrency (thousands of requests per minute). I will create an incident report to justify that point.

I believe this can be closed now (or we can wait for the incident report) and the ticket for restoring full functionality would be: T160983

I have written a preliminary version of the Incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20170320-Special_AllPages Feel free to correct/comment if you see any issues. After analyzing logs, I think the impact to users was not high, but it was creating important operational problems.

Hmm, this is the second time I've heard of using foo = '0' vs foo = 0 in queries adjusting the query plan. Maybe we should just have ints be ints in the db abstraction layer.

Should this issue be mentioned in Tech/News? Do you know this issue can be fixed before or after next Monday?

Reedy added a comment.Mar 23 2017, 5:31 PM

Should this issue be mentioned in Tech/News? Do you know this issue can be fixed before or after next Monday?

The page mostly isn't disabled anymore. Only the redirect filter is disabled...

The redirect filter certainly won't be coming back before monday

jcrespo closed this task as "Resolved".Mar 23 2017, 5:34 PM
jcrespo claimed this task.

@Trizek-WMF Reedy did a good summary of the current state at: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Special:AllPages_re-enabled This is resolved, but probably it would be nice to note the (at least, currently) disabled feature (filter by redirects). That followup is handled at: T160983