Page MenuHomePhabricator
Create Task
Maniphest T141523

Investigate web-05 downtime
Closed, ResolvedPublic
Actions

Description

icinga has been reporting minor downtime for ores-web-05 (but not ores-web-03). Investigate what is causing this.

Times in UTC from 2016-07-28:

[01:37:11] <icinga-wm> PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:41:11] <icinga-wm> RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 338 bytes in 9.751 second response time
[01:49:12] <icinga-wm> PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:53:02] <icinga-wm> RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 339 bytes in 1.546 second response time

Event Timeline

Halfak created this task.Jul 28 2016, 2:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2016, 2:12 PM
Halfak triaged this task as High priority.Jul 28 2016, 2:13 PM
Halfak moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.
Halfak added a project: ORES.
Halfak claimed this task.Jul 28 2016, 5:11 PM
Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

It looks like there was a spike in system CPU usage and a sudden drop in precache requests right before the downtime.

See:
https://graphite-labs.wikimedia.org/render/?width=586&height=308&_salt=1469725833.156&target=ores.ores-web-03.cpu.total.system&target=ores.ores-web-05.cpu.total.system&target=ores.ores-web-05.precache_request.count&target=ores.ores-web-03.precache_request.count

Halfak added a comment.Aug 2 2016, 9:48 PM

I thought I'd caught some new downtime today, but it turns out it was labs' proxy.

Halfak moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.Aug 4 2016, 9:48 PM

It looks like there's some OOM issues on web-05, so I reduced the amount of workers and it seems that we're doing better. I'm calling this done.

Ladsgroup closed this task as Resolved.Aug 8 2016, 9:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits