Page MenuHomePhabricator
Create Task
Maniphest T132845

API apache servers OOMing: mw1134 mw1132 mw1139 mw1138
Closed, DuplicatePublic
Actions

Description

This week four different app servers have OOMd and killed random processes:

mw1134
mw1132
mw1139
mw1138

I have rebooted all but 1138. 1138 I've depooled and left intact for investigation.

Event Timeline

Andrew created this task.Apr 16 2016, 2:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2016, 2:31 PM
Andrew updated the task description. (Show Details)Apr 16 2016, 2:34 PM
Andrew renamed this task from apache servers OOMing: mw1134 mw1132 mw1139 mw1138 to API apache servers OOMing: mw1134 mw1132 mw1139 mw1138.Apr 16 2016, 2:36 PM
Andrew triaged this task as Unbreak Now! priority.
Restricted Application added a subscriber: Urbanecm. · View Herald TranscriptApr 16 2016, 2:37 PM
Krenair subscribed.Apr 16 2016, 4:55 PM
Southparkfan subscribed.Apr 17 2016, 10:33 AM
Southparkfan added a comment.Apr 17 2016, 10:42 AM

@Andrew mw1138 is not depooled (anymore), its CPU and network graphs show it is serving traffic. Looking at http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=mw1132.eqiad.wmnet&m=cpu_report&s=by+name&mc=2&g=network_report&c=API+application+servers+eqiad it was idle for a few hours, but then it somehow got repooled..

Andrew lowered the priority of this task from Unbreak Now! to Medium.Apr 18 2016, 1:53 PM
ArielGlenn subscribed.Apr 22 2016, 5:06 PM
Andrew closed this task as a duplicate of T133674: HHVM is leaking memory on the API appservers.Apr 26 2016, 3:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL