Fatal error on Special:Log caused by date input (TimestampException error)
Closed, ResolvedPublic

Description

It seems someone is scanning wikis for SQL injection or other vulnerability related to improper string escaping. It is worth confirming that the code is not vulnerable.

The requests look like this:

/wiki/Special:Log?page=20&tagfilter=17&type=import&user=91.247.38.47&wpdate=%bf'%bf%22&wpfilters%5b%5d=review

https://logstash.wikimedia.org/goto/9f09650b6089006d15a8b47c4cebc245 (note this only captures the ones of this format, and not successful ones)

And we get errors such as:

Invalid timestamp - ¿'¿" 00:00:00
Invalid timestamp - @@GpY50 00:00:00
Invalid timestamp - JyI= 00:00:00
Invalid timestamp - (select convert(int,CHAR(65))) 00:00:00

Please have a look if this is blind testing with no impact and should be ignored, or if it is a known or 0 day exploitation (but clumsy execution).

Update: Probably the same person started later to do the same on enwiki, with a different function:

/w/index.php?associated=1&invert=1&namespace=-1;select%20pg_sleep(7);%20--%20&tagfilter=17&target=Main%20Page&title=Special:RecentChangesLinked   MWException from line 64 of /srv/mediawiki/php-1.32.0-wmf.15/includes/MWNamespace.php: MWNamespace::getAssociated does not make any sense for given namespace -1

And wikiquote before:

https://en.wikiquote.org/wiki/Special:Log?page=20&subtype=create&tagfilter=17&type=abusefilter&user=91.247.38.47&wpdate=-0001-11-30
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2018, 12:45 PM
Krinkle renamed this task from Automated injection scaning of en.wikibooks.org to Fatal ConvertibleTimestamp error on Special:Log (Automated injection scanning on en.wikibooks.org).Aug 7 2018, 1:14 PM
Krinkle edited projects, added MediaWiki-Logging; removed MediaWiki-Special-pages.
Krinkle moved this task from Untriaged to Meta on the Wikimedia-production-error board.
jcrespo renamed this task from Fatal ConvertibleTimestamp error on Special:Log (Automated injection scanning on en.wikibooks.org) to Automated injection scanning on several wikis.Aug 7 2018, 1:15 PM
jcrespo updated the task description. (Show Details)

One thing to note is that, even if there were no security issue here, this could be helpful as an externalized black box testing, and if we should check for the exception on code or otherwise return other kind of errors for the improper formatting.

This indeed persists and triggers "mediawiki exceptions" alerts due to high volume of attempts, eg.

[W2mdmgpAAEMAAB4PF5EAAAAK] /wiki/Special:Log?page=20&tagfilter=17&type=gblblock&user=91.247.38.47&wpdate=.%5c%5c./.%5c%5c./.%5c%5c./.%5c%5c./.%5c%5c./.%5c%5c./etc/passwd&wpfilters%5b%5d=tag   Wikimedia\Timestamp\TimestampException from line 147 of /srv/mediawiki/php-1.32.0-wmf.15/vendor/wikimedia/timestamp/src/ConvertibleTimestamp.php: Wikimedia\Timestamp\ConvertibleTimestamp::setTimestamp: Invalid timestamp - .\\./.\\./.\\./.\\./.\\./.\\./etc/passwd 00:00:00
jcrespo renamed this task from Automated injection scanning on several wikis to Automated injection scanning on several wikis ( Wikimedia\Timestamp\TimestampException and others).Aug 7 2018, 1:28 PM
jcrespo added a comment.EditedAug 7 2018, 1:33 PM

Most (if not all) seem to be coming from a single ip: https://logstash.wikimedia.org/goto/d57bb02a75a782d66b0a4fbf7d5c2199

Can also be tracked with the face user-agent, as it is not very frequent on errors: "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.21 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.21"

jcrespo updated the task description. (Show Details)Aug 7 2018, 1:52 PM
Krinkle added a subscriber: hoo.Aug 7 2018, 4:30 PM
Krinkle updated the task description. (Show Details)Aug 7 2018, 4:32 PM
Krinkle renamed this task from Automated injection scanning on several wikis ( Wikimedia\Timestamp\TimestampException and others) to Fatal error on Special:Log caused by date input (TimestampException error).Aug 7 2018, 4:37 PM
Krinkle triaged this task as High priority.
Krinkle removed a project: Security.
Krinkle updated the task description. (Show Details)
Krinkle changed the visibility from "Custom Policy" to "Public (No Login Required)".
Restricted Application added a project: Security. · View Herald TranscriptAug 7 2018, 4:37 PM
Krinkle added a subscriber: Krinkle.

HTTP 5xx with fatal error consistently reproducible with a url that can be created from normal user steps (not url hacking). The input is not a valid date, but typos can happen and it should be handled by Special:Log with an error displayed to the user, instead of going unhandled and bubbling to fatal level (thus alerting fatal-monitor and ops).

Krinkle added a comment.EditedSep 22 2018, 6:46 PM

This seems to be the cause of the MediaWiki Graphite alerts in #wikimedia-operations.

IRC
[17:12] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [50.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen
[17:36] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 70.00% above the threshold [25.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen
Grafana

Logstash
mediawiki-errorsmediawik-errors (this task's error only)

The wikis, servers, urls, and fields all vary. Which suggests this is either purely from user input by different people mistyping these dates, or a distributed attack intentionally aimed at overloading the servers.

In any event, this input is invalid, and showing the user an error is fine, but it needs to be an error served by MediaWiki's Special:Log. Instead of currently, where it bubbles up to the application layer and is logged as a crash of the software.

Change 462313 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462313

Change 462313 merged by jenkins-bot:
[mediawiki/core@master] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462313

Change 462526 had a related patch set uploaded (by Krinkle; owner: Legoktm):
[mediawiki/core@wmf/1.32.0-wmf.22] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462526

Krinkle assigned this task to Legoktm.Sep 25 2018, 2:39 AM
Krinkle closed this task as Resolved.

Deploying...

Change 462526 merged by jenkins-bot:
[mediawiki/core@wmf/1.32.0-wmf.22] SpecialLog: Don't throw exceptions on invalid date from user input

https://gerrit.wikimedia.org/r/462526

The IP 91.247.38.47 is still in the blacklist filters of the esams network devices, is it safe to remove it? If not, when should I ping you again about it?
Thanks

The IP 91.247.38.47 is still in the blacklist filters of the esams network devices, is it safe to remove it? If not, when should I ping you again about it?
Thanks

Yes, I think it should be fine to remove. I would note that it seems like that IP was doing some kind of sqli/scanning stuff, and I'm not sure whether want to allow them to resume that behavior. In any case, it should not cause any MediaWiki exceptions or alarms to trigger.

IP removed from Amsterdam router's blacklist.