Maniphest T140208

Write incident report for Content Translation downtime (20160712)
Closed, ResolvedPublic
Actions

Description

Incident: https://phabricator.wikimedia.org/T140123

Report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20160713-ContentTranslation

Related Objects

Mentioned In: T140865: Follow up work after the CX downtime incident
T140123: Content translation is bringing down x1-master
Mentioned Here: T140865: Follow up work after the CX downtime incident

Event Timeline

KartikMistry created this task.Jul 13 2016, 3:21 AM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 13 2016, 3:21 AM

• santhosh subscribed.Jul 13 2016, 3:36 AM

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160713-ContentTranslation

Nikerabbit added a project: Unplanned-Sprint-Work.Jul 13 2016, 9:11 AM

Looks good, @Nikerabbit

KartikMistry assigned this task to Nikerabbit.Jul 13 2016, 1:46 PM

KartikMistry moved this task from In Review to Done on the Language-Q1-2016-17 Sprint 1 board.

Language team has no monitoring in place that would have detected this issue.

What should we do about this?

Nikerabbit created subtask T140615: Decide on cxsave ping limiter limits and monitor.Jul 18 2016, 1:45 PM

Nikerabbit created subtask T140619: Contact the user who was causing saving spikes.Jul 18 2016, 1:57 PM

Nikerabbit created subtask T140622: Audit CX frontend code for possible causes for save spikes.Jul 18 2016, 2:03 PM

Nikerabbit mentioned this in T140123: Content translation is bringing down x1-master.Jul 19 2016, 8:18 AM

Amire80 moved this task from Needs Triage to CX10 (July-Sep. 2016) on the ContentTranslation board.Jul 20 2016, 6:12 AM

Arrbee mentioned this in T140865: Follow up work after the CX downtime incident.Jul 20 2016, 6:48 AM

Arrbee removed subtasks: T140622: Audit CX frontend code for possible causes for save spikes, T140619: Contact the user who was causing saving spikes, T140615: Decide on cxsave ping limiter limits and monitor.Jul 20 2016, 6:51 AM

Closing this ticket. Follow up work tracked via T140865

Amire80 moved this task from Backlog to High-priority Bugs on the ContentTranslation-Release10 board.Sep 7 2016, 12:16 PM

Restricted Application added a project: Language-Engineering July-September 2016. · View Herald TranscriptSep 7 2016, 12:16 PM

Amire80 moved this task from Backlog to CX10 on the Language-Engineering July-September 2016 board.Oct 9 2016, 12:22 PM