Page MenuHomePhabricator
Create Task
Maniphest T370401

cloudcontrol2006-dev struggling with memory
Closed, ResolvedPublic
Actions

Description

I saw an alert today:

FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM

The logs showed that the user with id 111 got a bunch of procs killed by the OOMkiller. This is rabbitmq.

The system had a very high load average as well.

I noticed the physical memory in the system is low (32GB) compared to other similar servers (cloudcontrol2004-dev 128GB, cloudcontrol2005-dev 500GB).

This server is definitely under memory pressure, on a normal day it will have only 1GB available.

Related Objects
Search...

Event Timeline

aborrero created this task.Jul 18 2024, 10:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 18 2024, 10:01 AM
aborrero added a comment.Jul 18 2024, 10:49 AM

the alert triggered again.

joanna_borun triaged this task as Medium priority.Aug 21 2024, 2:17 PM
aborrero changed the task status from Open to In Progress.Oct 18 2024, 10:50 AM
aborrero raised the priority of this task from Medium to High.
aborrero added a project: User-aborrero.

Today the oomkiller victim was mariadb, which is maybe even more concerning that rabbitmq getting killed.

I'll raise priority, and see if DCops can help us replace this server.

aborrero added a parent task: Unknown Object (Task).Oct 18 2024, 11:07 AM
aborrero mentioned this in T377568: wmcs codfw hardware changes proposal.Oct 18 2024, 12:04 PM
taavi added a project: Cloud-VPS.Oct 22 2024, 5:36 PM
aborrero assigned this task to Papaul.Oct 23 2024, 10:18 AM
aborrero added a project: ops-codfw.
aborrero moved this task from Backlog to Blocked/waiting on the User-aborrero board.
aborrero added subscribers: Jhancock.wm, Papaul.

hey @Papaul and/or @Jhancock.wm per https://phabricator.wikimedia.org/T377568#10247882 you should be receiving memory expansion parts soon. Please upgrade the memory on this server to 128GB.

Restricted Application added a project: DC-Ops. · View Herald TranscriptOct 23 2024, 10:18 AM
aborrero removed a project: Cloud-VPS.Oct 23 2024, 10:19 AM
Maintenance_bot added a project: SRE.Oct 23 2024, 10:29 AM
Jhancock.wm moved this task from Backlog to Racking Tasks on the ops-codfw board.Oct 23 2024, 2:01 PM
Jhancock.wm moved this task from Racking Tasks to Hardware Failure / Troubleshoot on the ops-codfw board.
Jhancock.wm added a comment.Wed, Oct 30, 2:06 PM

I got the memory in. Is it safe to proceed with the upgrade at this time? I didn't see if it got depooled already.

aborrero added a comment.Wed, Oct 30, 4:20 PM
In T370401#10276958, @Jhancock.wm wrote:

I got the memory in. Is it safe to proceed with the upgrade at this time? I didn't see if it got depooled already.

thanks @Jhancock.wm, the server can be upgraded anytime.

Jhancock.wm closed this task as Resolved.Fri, Nov 1, 2:57 PM
Jhancock.wm claimed this task.

upgraded to 128Gb. powering up now. Please let us know if you need additional help!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits