Special:AllPages disabled due to performance issues
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Reedy
	Mar 20 2017, 3:17 PM

Description

After Special:AllPages causing an outage earlier today, it has been temporarily disabled pending investigation.

This ticket serves as the public tracking for the issue

Details

	Project	Branch	Lines +/-	Subject
	mediawiki/core	master	+46 -6	Disable filter by redirect Special:AllPages and query=allpages in miser mode
	mediawiki/core	wmf/1.29.0-wmf.16	+46 -6	Disable filter by redirect Special:AllPages and query=allpages in miser mode

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	jcrespo	T160916 Special:AllPages disabled due to performance issues
Resolved	Reedy	T160914 Databases overflown with connections due to slow query on Special:AllPages
Open	None	T160920 Throttling access to Special Pages that make potentially expensive queries

Event Timeline

Reedy created this task.Mar 20 2017, 3:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2017, 3:17 PM

Reedy added a subtask: T160914: Databases overflown with connections due to slow query on Special:AllPages.Mar 20 2017, 3:17 PM

Umherirrender added a subscriber: Umherirrender.Mar 20 2017, 3:28 PM

Should be high priority as it affects prod wiki's.

stwalkerster added a subscriber: stwalkerster.Mar 20 2017, 3:30 PM

Jdforrester-WMF added a subscriber: Jdforrester-WMF.Mar 20 2017, 3:31 PM

Special:Allpages still displays MediaWiki:Allpages-summary at the top so wikis can create the message to explain it like I did in https://en.wikipedia.org/wiki/MediaWiki:Allpages-summary and mentioned in https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Special:AllPages_disabled. I guess a temporary MediaWiki default for MediaWiki:Allpages-summary might not be displayed before the issue is fixed.

revi added a subscriber: revi.Mar 20 2017, 3:43 PM

JEumerus added a subscriber: JEumerus.Mar 20 2017, 3:46 PM

zhuyifei1999 added a subscriber: zhuyifei1999.Mar 20 2017, 3:50 PM

Betacommand added a subscriber: Betacommand.Mar 20 2017, 3:55 PM

hashar added a subscriber: hashar.Mar 20 2017, 3:59 PM

Mentioned in SAL (#wikimedia-operations) [2017-03-20T15:59:51Z] <hashar> Special:AllPages being blank has a public task: https://phabricator.wikimedia.org/T160916

Reedy created subtask T160920: Throttling access to Special Pages that make potentially expensive queries.Mar 20 2017, 4:08 PM

jcrespo added a project: User-notice.Mar 20 2017, 4:30 PM

Thibaut120094 added a subscriber: Thibaut120094.Mar 20 2017, 4:59 PM

KTC added a subscriber: KTC.Mar 20 2017, 5:05 PM

Samuele2002 added a subscriber: Samuele2002.Mar 20 2017, 5:11 PM

• MZMcBride added a subscriber: • MZMcBride.Mar 20 2017, 5:20 PM

Vituzzu added a subscriber: Vituzzu.Mar 20 2017, 5:50 PM

Change 343681 had a related patch set uploaded (by Reedy):
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343681

gerritbot added a project: Patch-For-Review.Mar 20 2017, 5:55 PM

I think it can safely re-enabled on significantly smaller wikis even prior to any investigation. Or better, IMHO, it can be disabled just on a limited set of bigger wikis.

In T160916#3115185, @Vituzzu wrote:

I think it can safely re-enabled on significantly smaller wikis even prior to any investigation. Or better, IMHO, it can be disabled just on a limited set of bigger wikis.

Hence the patch to disable the offending feature in miser mode :)

Matthewrbowker added a subscriber: Matthewrbowker.Mar 20 2017, 6:44 PM

Sorry :(

Samuele2002 added subscribers: Matthewrbowker, Vituzzu, GerritBot and 3 others.Mar 20 2017, 6:52 PM

Vituzzu added a project: Patch-For-Review.Mar 20 2017, 6:54 PM

Vituzzu added subscribers: • MZMcBride, hashar, Betacommand and 3 others.

Sorry. They are so mobile and when I added has removed everything else. It already happened once when they are mobile. someone tell me why?

Samuele2002 added a subscriber: • MZMcBride.Mar 20 2017, 7:02 PM

Change 343711 had a related patch set uploaded (by Reedy):
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343711

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

In T160916#3115802, @JEumerus wrote:

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

We had a user hitting enwiki enmass screenscraping which caused issues.

In T160916#3115823, @Betacommand wrote:

In T160916#3115802, @JEumerus wrote:

Hum. Why did performance issues occur just now? Special:AllPages has been around for a long time.

We had a user hitting enwiki enmass screenscraping which caused issues.

Well, that and there's potentially some issues in MariaDB's query planner that we may be experiencing post upgrades too

Change 343681 merged by jenkins-bot:
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343681

Change 343711 merged by jenkins-bot:
[mediawiki/core] Disable filter by redirect Special:AllPages and query=allpages in miser mode

https://gerrit.wikimedia.org/r/343711

Reedy removed a project: Patch-For-Review.Mar 20 2017, 8:11 PM

Mentioned in SAL (#wikimedia-operations) [2017-03-20T20:12:21Z] <reedy@tin> Synchronized php-1.29.0-wmf.16/includes/specials/SpecialAllPages.php: Re-enable Special:AllPages, disable redirect filter if MiserMode T160916 (duration: 00m 42s)

Mentioned in SAL (#wikimedia-operations) [2017-03-20T20:14:12Z] <reedy@tin> Synchronized php-1.29.0-wmf.16/includes/api/ApiQueryAllPages.php: Limit query=allpages filterredir if MiserMode T160916 (duration: 00m 42s)

tom29739 added a subscriber: tom29739.Mar 20 2017, 11:28 PM

jcrespo closed subtask T160914: Databases overflown with connections due to slow query on Special:AllPages as Resolved.Mar 21 2017, 10:27 AM

there's potentially some issues in MariaDB's query planner

While that is probably true, I believe that the root cause was a not perfect query, in some very specific cases, in some very large wikis, having bad performance- something that only was surfaced when someone tried to scrap all pages with very high concurrency (thousands of requests per minute). I will create an incident report to justify that point.

I believe this can be closed now (or we can wait for the incident report) and the ticket for restoring full functionality would be: T160983

Anomie mentioned this in T160983: Optimize SpecialAllPages::showChunk for large wikis.Mar 21 2017, 2:11 PM

I have written a preliminary version of the Incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20170320-Special_AllPages Feel free to correct/comment if you see any issues. After analyzing logs, I think the impact to users was not high, but it was creating important operational problems.

ReleaseTaggerBot added projects: MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), MW-1.29-release-notes.Mar 21 2017, 6:01 PM

Hmm, this is the second time I've heard of using foo = '0' vs foo = 0 in queries adjusting the query plan. Maybe we should just have ints be ints in the db abstraction layer.

Should this issue be mentioned in Tech/News? Do you know this issue can be fixed before or after next Monday?

In T160916#3125669, @Trizek-WMF wrote:

Should this issue be mentioned in Tech/News? Do you know this issue can be fixed before or after next Monday?

The page mostly isn't disabled anymore. Only the redirect filter is disabled...

The redirect filter certainly won't be coming back before monday

@Trizek-WMF Reedy did a good summary of the current state at: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Special:AllPages_re-enabled This is resolved, but probably it would be nice to note the (at least, currently) disabled feature (filter by redirects). That followup is handled at: T160983