Page MenuHomePhabricator

Content translation is bringing down x1-master
Closed, ResolvedPublic

Event Timeline

jcrespo created this task.Jul 12 2016, 3:59 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 12 2016, 3:59 PM
jcrespo triaged this task as Unbreak Now! priority.Jul 12 2016, 3:59 PM
Restricted Application added subscribers: Luke081515, TerraCodes. · View Herald TranscriptJul 12 2016, 3:59 PM

Change 298510 had a related patch set uploaded (by Chad):
Disable content translation

https://gerrit.wikimedia.org/r/298510

It had consumed all connections (the 10000 of them) doing:

UPDATE /* ContentTranslation\Translation::update <user> */  `cx_translations` SET translation_target_title = 'Doaba', translation_source_revision_id = '729218022', translation_source_url = '//en.wikipedia.org/wiki/Doaba', translation_status = 'draft', translation_last_updated_timestamp = '20160712153934', translation_progress = '{\"any\":0.4634964890541099, \"human\":0.3580132176786452, \"mt\":0.10548327137546469, \"mtSectionsCount\":16}', translation_last_update_by = '42573130' WHERE translation_id = '177521' /* da58dc0b53791be4d3bda4ef4224b952 db1031 wikishared 22s */

Change 298510 merged by Chad:
Disable content translation

https://gerrit.wikimedia.org/r/298510

greg added a subscriber: greg.Jul 12 2016, 4:15 PM
Arrbee added a subscriber: Arrbee.Jul 12 2016, 4:27 PM

Change 298522 had a related patch set uploaded (by Nikerabbit):
Add ping limiter to cxsave

https://gerrit.wikimedia.org/r/298522

Change 298523 had a related patch set uploaded (by Nikerabbit):
Add rate limiting for cxsave

https://gerrit.wikimedia.org/r/298523

API module request counts shows spikes of over 1000 (per second I assume), assumed to come from one or few users.

Change 298523 merged by jenkins-bot:
Add rate limiting for cxsave

https://gerrit.wikimedia.org/r/298523

Change 298522 merged by jenkins-bot:
Add ping limiter to cxsave

https://gerrit.wikimedia.org/r/298522

Change 298529 had a related patch set uploaded (by Chad):
Add ping limiter to cxsave

https://gerrit.wikimedia.org/r/298529

Change 298529 merged by jenkins-bot:
Add ping limiter to cxsave

https://gerrit.wikimedia.org/r/298529

Nikerabbit closed this task as Resolved.Jul 19 2016, 8:18 AM
Nikerabbit claimed this task.

No new spikes, and protection is in place. Incident report has been written (T140208). Closing.