Page MenuHomePhabricator

Special:AllPages disabled due to performance issues
Closed, ResolvedPublic

Description

After Special:AllPages causing an outage earlier today, it has been temporarily disabled pending investigation.

This ticket serves as the public tracking for the issue

Event Timeline

Paladox triaged this task as High priority.Mar 20 2017, 3:29 PM
Paladox added a subscriber: Paladox.

Should be high priority as it affects prod wiki's.

Special:Allpages still displays MediaWiki:Allpages-summary at the top so wikis can create the message to explain it like I did in https://en.wikipedia.org/wiki/MediaWiki:Allpages-summary and mentioned in https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Special:AllPages_disabled. I guess a temporary MediaWiki default for MediaWiki:Allpages-summary might not be displayed before the issue is fixed.

Change 343681 had a related patch set uploaded (by Reedy):
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343681

I think it can safely re-enabled on significantly smaller wikis even prior to any investigation. Or better, IMHO, it can be disabled just on a limited set of bigger wikis.

I think it can safely re-enabled on significantly smaller wikis even prior to any investigation. Or better, IMHO, it can be disabled just on a limited set of bigger wikis.

Hence the patch to disable the offending feature in miser mode :)

Sorry. They are so mobile and when I added has removed everything else. It already happened once when they are mobile. someone tell me why?

Change 343711 had a related patch set uploaded (by Reedy):
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343711

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

We had a user hitting enwiki enmass screenscraping which caused issues.

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

We had a user hitting enwiki enmass screenscraping which caused issues.

Well, that and there's potentially some issues in MariaDB's query planner that we may be experiencing post upgrades too

Change 343681 merged by jenkins-bot:
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343681

Change 343711 merged by jenkins-bot:
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343711

Mentioned in SAL (#wikimedia-operations) [2017-03-20T20:12:21Z] <reedy@tin> Synchronized php-1.29.0-wmf.16/includes/specials/SpecialAllPages.php: Re-enable Special:AllPages, disable redirect filter if MiserMode T160916 (duration: 00m 42s)

Mentioned in SAL (#wikimedia-operations) [2017-03-20T20:14:12Z] <reedy@tin> Synchronized php-1.29.0-wmf.16/includes/api/ApiQueryAllPages.php: Limit query=allpages filterredir if MiserMode T160916 (duration: 00m 42s)

there's potentially some issues in MariaDB's query planner

While that is probably true, I believe that the root cause was a not perfect query, in some very specific cases, in some very large wikis, having bad performance- something that only was surfaced when someone tried to scrap all pages with very high concurrency (thousands of requests per minute). I will create an incident report to justify that point.

I believe this can be closed now (or we can wait for the incident report) and the ticket for restoring full functionality would be: T160983

I have written a preliminary version of the Incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20170320-Special_AllPages Feel free to correct/comment if you see any issues. After analyzing logs, I think the impact to users was not high, but it was creating important operational problems.

Hmm, this is the second time I've heard of using foo = '0' vs foo = 0 in queries adjusting the query plan. Maybe we should just have ints be ints in the db abstraction layer.

Should this issue be mentioned in Tech/News? Do you know this issue can be fixed before or after next Monday?

Should this issue be mentioned in Tech/News? Do you know this issue can be fixed before or after next Monday?

The page mostly isn't disabled anymore. Only the redirect filter is disabled...

The redirect filter certainly won't be coming back before monday

jcrespo claimed this task.

@Trizek-WMF Reedy did a good summary of the current state at: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Special:AllPages_re-enabled This is resolved, but probably it would be nice to note the (at least, currently) disabled feature (filter by redirects). That followup is handled at: T160983