Page MenuHomePhabricator

Fatal InvalidArgumentException on various Special:AbuseLog urls
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error

Request URL: https://en.wikipedia.org/w/index.php?title=Special:AbuseLog&wpSearchFilter=1%27%20UNION
Request ID: XRtuagpAMFAAAGpCaTMAAABP

message
InvalidArgumentException: Invalid filter name (gibberish)
trace
#0 /srv/mediawiki/php-1.34.0-wmf.11/extensions/AbuseFilter/includes/AbuseFilter.php(373): AbuseFilter::splitGlobalName(string)
#1 /srv/mediawiki/php-1.34.0-wmf.11/extensions/AbuseFilter/includes/special/SpecialAbuseLog.php(466): AbuseFilter::filterHidden(string)
#2 /srv/mediawiki/php-1.34.0-wmf.11/extensions/AbuseFilter/includes/special/SpecialAbuseLog.php(130): SpecialAbuseLog->showList()
#3 /srv/mediawiki/php-1.34.0-wmf.11/includes/specialpage/SpecialPage.php(571): SpecialAbuseLog->execute(NULL)
#4 /srv/mediawiki/php-1.34.0-wmf.11/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#5 /srv/mediawiki/php-1.34.0-wmf.11/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#6 /srv/mediawiki/php-1.34.0-wmf.11/includes/MediaWiki.php(884): MediaWiki->performRequest()
#7 /srv/mediawiki/php-1.34.0-wmf.11/includes/MediaWiki.php(515): MediaWiki->main()
#8 /srv/mediawiki/php-1.34.0-wmf.11/index.php(42): MediaWiki->run()
#9 /srv/mediawiki/w/index.php(3): include(string)
#10 {main}

Impact

Users can craft urls to Special:AbuseLog that consistently cause a bypass of the Varnish cache, hit the application servers, and make them generate a fatal error.

These fatal errors then produce an HTTP 500 response which raises alert levels for the Traffic layers, as well as deployment health in Logstash which can cause alerts and aborted deployments.

Notes

This appears new in 1.34-wmf.10/1.34-wmf.11. It was rarely seen or not at all, in the 3 weeks before that.

Event Timeline

Yeah, I was just looking at this. My conclusion is that it's due to the validation introduced in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/488401/. However, I also see that all of those requests are obviously malicious, and probably people have been running them forever. It's just that now we throw before even querying the DB.
While the best solution would be to adopt countermeasures against this type of requests, I see that https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/499883/ will fix the error by catching the exception. The patch was already +2ed but not merged because of a merge conflict. I'm going to fix it, and @Krinkle would you be fine with rechecking it once done?

Change 499883 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Tweak methods related to global filters

https://gerrit.wikimedia.org/r/499883

@Daimona Yep, will do. Can you also prepare a cherry-pick compatible with wmf.11 for deployment?

Change 520256 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@wmf/1.34.0-wmf.11] Tweak methods related to global filters

https://gerrit.wikimedia.org/r/520256

Change 520256 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@wmf/1.34.0-wmf.11] Tweak methods related to global filters

https://gerrit.wikimedia.org/r/520256

Here it is, nothing new touched that code so everything should be fine.

Change 499883 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Tweak methods related to global filters

https://gerrit.wikimedia.org/r/499883

Change 520256 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@wmf/1.34.0-wmf.11] Tweak methods related to global filters

https://gerrit.wikimedia.org/r/520256

Daimona removed a project: Patch-For-Review.

Mentioned in SAL (#wikimedia-operations) [2019-07-02T19:05:30Z] <krinkle@deploy1001> Synchronized php-1.34.0-wmf.11/extensions/AbuseFilter/: rEABF9963d843622b / T227095 (duration: 00m 51s)

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM