Page MenuHomePhabricator

Monitor any sql syntax error issues, and send email alerting if they happen
Open, MediumPublic

Description

As a guard against SQL injection attacks, we should monitor synatx error DB query exceptions, and send an email notice to security-team@wikimedia.org if they suddenly start happening.

I'm not very famialr with monitoring infastructure at wikimedia, but maybe incinga could be used to check https://logstash.wikimedia.org/goto/eed39a6fab525646bec815b27be3bed0 on a regular basis.

Event Timeline

Bawolff triaged this task as Medium priority.Oct 4 2017, 5:08 PM
Bawolff added a subscriber: jcrespo.

Cool with this, although I will warn you about potential horrible deployments, so make sure you don't do a DOS to your mail or get too many false positicces :-)

If performance_schema configuration/archiving is fixed T164834, there is an "errors and warnings" view on server side that may be more efficient than mediawiki itself. Mediawiki, of course, is cleaner and easier (syntax errors have its own error code). Just as an idea as a monitoring thing for both performance and security "new query patterns found".

root@db1053[sys]> SELECT * FROM statements_with_errors_or_warnings;
+-------------------------------------------------------------------+--------------------+------------+--------+-----------+---------
| query                                                             | db                 | exec_count | errors | error_pct | warnings
+-------------------------------------------------------------------+--------------------+------------+--------+-----------+---------
| SELECT `rc_id` , `rc_timestamp ... Y `rc_timestamp` DESC LIMIT ?  | commonswiki        |     461361 |   1257 |    0.2725 |        0

root@db1053[sys]> SELECT * FROM statement_analysis;
+-------------------------------------------------------------------+--------------------+-----------+------------+-----------+------
| query                                                             | db                 | full_scan | exec_count | err_count | warn_
+-------------------------------------------------------------------+--------------------+-----------+------------+-----------+------
| SELECT `MASTER_GTID_WAIT` (...)                                   | NULL               |           |  359785541 |         0 |

Re:

I'm not very famialr with monitoring infastructure at wikimedia, but maybe incinga could be used to check

Probably the best way is to send numbers to prometheus with a custom exporter and create alarms based on numbers (it can connect to icinga, too, as we do with performance numbers)? The biggest issue is avoid false positives. Nobody wants to receive alerts because someone wrote bad code non-maliciously.

E.g: https://grafana.wikimedia.org/dashboard/db/navigation-timing-alerts?refresh=5m&panelId=13&fullscreen&orgId=1