Msnbot hit yesterday night between 10:07 UTC and performed >500 requests like:
```
(SELECT /* SpecialRecentChangesLinked::doMainQuery <IP> */ rc_id, rc_timestamp, rc_user, rc_user_text, rc_namespace, rc_title, rc_comment, rc_minor, rc_bot, rc_new, rc_cur_id, rc_this_oldid, rc_last_oldid, rc_type, rc_source, rc_patrolled, rc_ip, rc_old_len, rc_new_len, rc_deleted, rc_logid, rc_log_type, rc_log_action, rc_params, (SELECT GROUP_CONCAT(ct_tag SEPARATOR ', ') FROM `change_tag` WHERE ct_rc_id=rc_id ) AS `ts_tags` FROM `recentchanges` INNER JOIN `pagelinks` ON ((rc_namespace = pl_namespace) AND (rc_title = pl_title)) WHERE rc_bot = '0' AND (rc_type != '6') AND (rc_timestamp >= '20160503000000') AND (rc_source != 'wb') AND pl_from = '1279706' ORDER BY rc_timestamp DESC LIMIT 50 ) UNION (SELECT rc_id, rc_timestamp, rc_user, rc_user_text, rc_namespace, rc_title, rc_comment, rc_minor, rc_bot, rc_new, rc_cur_id, rc_this_oldid, rc_last_oldid, rc_type, rc_source, rc_patrolled, rc_ip, rc_old_len, rc_new_len, rc_deleted, rc_logid, rc_log_type, rc_log_action, rc_params, (SELECT GROUP_CONCAT(ct_tag SEPARATOR ', ') FROM `change_tag` WHERE ct_rc_id=rc_id ) AS `ts_tags` FROM `recentchanges` INNER JOIN `templatelinks` ON ((rc_namespace = tl_namespace) AND (rc_title = tl_title)) WHERE rc_bot = '0' AND (rc_type != '6') AND (rc_timestamp >= '20160503000000') AND (rc_source != 'wb') AND tl_from = '1279706' ORDER BY rc_timestamp DESC LIMIT 50 ) UNION (SELECT rc_id, rc_timestamp, rc_user, rc_user_text, rc_namespace, rc_title, rc_comment, rc_minor, rc_bot, rc_new, rc_cur_id, rc_this_oldid, rc_last_oldid, rc_type, rc_source, rc_patrolled, rc_ip, rc_old_len, rc_new_len, rc_deleted, rc_logid, rc_log_type, rc_log_action, rc_params, (SELECT GROUP_CONCAT(ct_tag SEPARATOR ', ') FROM `change_tag` WHERE ct_rc_id=rc_id ) AS `ts_tags` FROM `recentchanges` INNER JOIN `imagelinks` ON ((rc_title = il_to)) WHERE rc_bot = '0' AND (rc_type != '6') AND (rc_timestamp >= '20160503000000') AND (rc_source != 'wb') AND il_from = '1279706' AND rc_namespace = '6' ORDER BY rc_timestamp DESC LIMIT 50 ) ORDER BY rc_timestamp DESC LIMIT 50
```
that took between 20 and 40 seconds, exhausting the number of connections available, creating lag and 5000-10000 failed requests:
{F3994364}
https://logstash.wikimedia.org/#dashboard/temp/AVSfBVO70z-7ykXOX4eb
I am only 99% sure this is the cause, there were other issues happening at the same time, but I think much less impacting: ApiQueryContributions::execute, which I believe it is already a known, but minor issue.
I am pinging releng because I believe this could be related to recent code deploy and potentially cause worse issues if more widely deployed, so I am asking to:
a) pause the train to analyze if new code could be responsible of that (I want to remember that RecentChangesLinked is a new functionality, with some issues in the past)
b) know who in mediawiki is responsible for it and assign the appropriate tags (this is of course not an issue for you to directly manage)
c) Check if robots.txt is correct, and bounce back to operations if needed