Page MenuHomePhabricator
Create Task
Maniphest T148652

2016-10-17 API cluster overload
Closed, ResolvedPublic
Actions

Description

This task tracks the investigation of a partial outage of the MediaWiki API cluster, which started at 15:18 on 2016-10-17 and ended at XXX.

What happened:

  • Requests piled up on individual app servers.
  • App servers eventually became unresponsive and were depooled.
  • The number of pooled API backends shrank until it could no longer meet demand
  • Users were served 5xx errors.

Related Objects

Event Timeline

ori created this task.Oct 19 2016, 3:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 19 2016, 3:25 PM
Joe subscribed.Oct 19 2016, 3:26 PM
ori added a comment.EditedOct 19 2016, 3:32 PM

Backtrace from mw1194: P4264

greg added a project: Wikimedia-Incident.Oct 19 2016, 7:19 PM
greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
elukey triaged this task as High priority.Oct 20 2016, 1:30 PM
daniel subscribed.Oct 21 2016, 1:07 PM
fgiunchedi mentioned this in T151702: API cluster failure / OOM.Nov 28 2016, 8:27 PM
greg added subscribers: fgiunchedi, greg.Feb 13 2017, 10:22 PM

So, this never happened (this == the investigation). It is now almost 4 months later, with a new year in between. There was no incident report written and nothing obvious from SAL: https://tools.wmflabs.org/sal/production?p=0&q=&d=2016-10-17

This task was referenced in T151702#2828577.

In T148652#2729028, @ori wrote:

Backtrace from mw1194: P4264

I can't view that paste :/

@fgiunchedi thoughts on whether we should keep this task open any longer? Anything you think we can still do at this point?

fgiunchedi closed this task as Resolved.Feb 14 2017, 1:11 PM
fgiunchedi claimed this task.

@greg I think it is safe to close, we've mitigated the issue by having a separate cluster for async processing in T151702 and related.

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits