Page MenuHomePhabricator

Fatal error on Special:Log caused by date input (TimestampException error)
Closed, ResolvedPublicPRODUCTION ERROR

Description

It seems someone is scanning wikis for SQL injection or other vulnerability related to improper string escaping. It is worth confirming that the code is not vulnerable.

The requests look like this:

/wiki/Special:Log?page=20&tagfilter=17&type=import&user=91.247.38.47&wpdate=%bf'%bf%22&wpfilters%5b%5d=review

https://logstash.wikimedia.org/goto/9f09650b6089006d15a8b47c4cebc245 (note this only captures the ones of this format, and not successful ones)

And we get errors such as:

Invalid timestamp - ¿'¿" 00:00:00
Invalid timestamp - @@GpY50 00:00:00
Invalid timestamp - JyI= 00:00:00
Invalid timestamp - (select convert(int,CHAR(65))) 00:00:00

Please have a look if this is blind testing with no impact and should be ignored, or if it is a known or 0 day exploitation (but clumsy execution).

Update: Probably the same person started later to do the same on enwiki, with a different function:

/w/index.php?associated=1&invert=1&namespace=-1;select%20pg_sleep(7);%20--%20&tagfilter=17&target=Main%20Page&title=Special:RecentChangesLinked   MWException from line 64 of /srv/mediawiki/php-1.32.0-wmf.15/includes/MWNamespace.php: MWNamespace::getAssociated does not make any sense for given namespace -1

And wikiquote before:

https://en.wikiquote.org/wiki/Special:Log?page=20&subtype=create&tagfilter=17&type=abusefilter&user=91.247.38.47&wpdate=-0001-11-30

Event Timeline

Krinkle renamed this task from Automated injection scaning of en.wikibooks.org to Fatal ConvertibleTimestamp error on Special:Log (Automated injection scanning on en.wikibooks.org).Aug 7 2018, 1:14 PM
Krinkle edited projects, added MediaWiki-Logevents; removed MediaWiki-Special-pages.
Krinkle moved this task from Untriaged to Mar 2021 on the Wikimedia-production-error board.
jcrespo renamed this task from Fatal ConvertibleTimestamp error on Special:Log (Automated injection scanning on en.wikibooks.org) to Automated injection scanning on several wikis.Aug 7 2018, 1:15 PM
jcrespo updated the task description. (Show Details)

One thing to note is that, even if there were no security issue here, this could be helpful as an externalized black box testing, and if we should check for the exception on code or otherwise return other kind of errors for the improper formatting.

This indeed persists and triggers "mediawiki exceptions" alerts due to high volume of attempts, eg.

[W2mdmgpAAEMAAB4PF5EAAAAK] /wiki/Special:Log?page=20&tagfilter=17&type=gblblock&user=91.247.38.47&wpdate=.%5c%5c./.%5c%5c./.%5c%5c./.%5c%5c./.%5c%5c./.%5c%5c./etc/passwd&wpfilters%5b%5d=tag   Wikimedia\Timestamp\TimestampException from line 147 of /srv/mediawiki/php-1.32.0-wmf.15/vendor/wikimedia/timestamp/src/ConvertibleTimestamp.php: Wikimedia\Timestamp\ConvertibleTimestamp::setTimestamp: Invalid timestamp - .\\./.\\./.\\./.\\./.\\./.\\./etc/passwd 00:00:00
jcrespo renamed this task from Automated injection scanning on several wikis to Automated injection scanning on several wikis ( Wikimedia\Timestamp\TimestampException and others).Aug 7 2018, 1:28 PM

Most (if not all) seem to be coming from a single ip: https://logstash.wikimedia.org/goto/d57bb02a75a782d66b0a4fbf7d5c2199

Can also be tracked with the face user-agent, as it is not very frequent on errors: "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.21 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.21"

Krinkle renamed this task from Automated injection scanning on several wikis ( Wikimedia\Timestamp\TimestampException and others) to Fatal error on Special:Log caused by date input (TimestampException error).Aug 7 2018, 4:37 PM
Krinkle triaged this task as High priority.
Krinkle removed a project: acl*security.
Krinkle updated the task description. (Show Details)
Krinkle changed the visibility from "Custom Policy" to "Public (No Login Required)".
Krinkle subscribed.

HTTP 5xx with fatal error consistently reproducible with a url that can be created from normal user steps (not url hacking). The input is not a valid date, but typos can happen and it should be handled by Special:Log with an error displayed to the user, instead of going unhandled and bubbling to fatal level (thus alerting fatal-monitor and ops).

This seems to be the cause of the MediaWiki Graphite alerts in #wikimedia-operations.

IRC

[17:12] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [50.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen
[17:36] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 70.00% above the threshold [25.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen

Grafana

Screen Shot 2018-09-22 at 19.38.14.png (602×2 px, 146 KB)

Logstash
mediawiki-errorsmediawik-errors (this task's error only)
Screen Shot 2018-09-22 at 19.38.10.png (1×2 px, 482 KB)
Screen Shot 2018-09-22 at 19.38.10.png (602×2 px, 107 KB)

The wikis, servers, urls, and fields all vary. Which suggests this is either purely from user input by different people mistyping these dates, or a distributed attack intentionally aimed at overloading the servers.

In any event, this input is invalid, and showing the user an error is fine, but it needs to be an error served by MediaWiki's Special:Log. Instead of currently, where it bubbles up to the application layer and is logged as a crash of the software.

Change 462313 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462313

Change 462313 merged by jenkins-bot:
[mediawiki/core@master] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462313

Change 462526 had a related patch set uploaded (by Krinkle; owner: Legoktm):
[mediawiki/core@wmf/1.32.0-wmf.22] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462526

Krinkle assigned this task to Legoktm.

Deploying...

Change 462526 merged by jenkins-bot:
[mediawiki/core@wmf/1.32.0-wmf.22] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462526

The IP 91.247.38.47 is still in the blacklist filters of the esams network devices, is it safe to remove it? If not, when should I ping you again about it?
Thanks

The IP 91.247.38.47 is still in the blacklist filters of the esams network devices, is it safe to remove it? If not, when should I ping you again about it?
Thanks

Yes, I think it should be fine to remove. I would note that it seems like that IP was doing some kind of sqli/scanning stuff, and I'm not sure whether want to allow them to resume that behavior. In any case, it should not cause any MediaWiki exceptions or alarms to trigger.

IP removed from Amsterdam router's blacklist.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM