A couple of things that we discussed with the team:

• this is the same problem that happens on stat machines, sometimes users are not conservative in their usage of those hosts consuming all the resources and triggering the OOM.
• the nagios server could have a adjusted score for the OOM to avoid being killed when it is reclaiming memory (probably not really feasible to do).
• Basic ulimits could prevent a process to eat up all the memory (this is usually the main issue), we could start with it and see how it goes.

I am pretty ignorant about it, but would cgroups fit in this use case? @MoritzMuehlenhoff ?

Just as an extra data point, early morning 2019-01-22 nagios-nrpe-server crashed on stat1007 from a cannot allocate error.

Mentioned in SAL (#wikimedia-operations) [2019-01-24T18:53:15Z] <mutante> notebook1003 - restarted nagios-nrpe-server... T212824

Mentioned in SAL (#wikimedia-operations) [2019-01-25T17:17:02Z] <chaomodus> notebook1003 restarted nagios-nrpe-server due to oom - T212824

Thanks a lot for all the work :)

I am leaning towards having a cgroup that limits CPU/memory usage of all processes to something like 80/85%, so hopefully this will give people the ability to crunch data without risking to trigger the OOM killer. I need to document and learn how to properly apply cgroups (and not sure if we have anything generic/ready-to-use in puppet).

Quick idea that could alleviate this issue: if we create a cgroup like https://wiki.archlinux.org/index.php/cgroups#Matlab on the stat/notebook hosts, and ask the users to either run their script with something like cgexec -g memory,cpuset:matlab prepended or via a wrapper (like `wmf-group-exec mycommand -etc..) we could place all computations in a cgroup that can maximum consume 80/85% of ram, avoiding to saturate the host. We could show up a warning in motd to users that a cgroup needs to be used for computations, and follow up with whoever doesn't do it.

After a chat with Chase, it seems that @aborrero has already done a similar thing for the tool-forge hosts. Anything that we can share with puppet? Ideally in my use case if a user belongs to certain groups (say analytics-privatedata-users, etc..) its login session + processes should all go into a cgroup (create beforehand) that limits the memory usage to something like 85% of the overall (to leave space for OS daemons etc.. and possibly avoid the OOM killer parties). So basically having users of certain groups to create processes under the same cgroup, limited in memory usage (and maybe cpu cores used?).

In T212824#4922713, @elukey wrote:

@aborrero has already done a similar thing for the tool-forge hosts. Anything that we can share with puppet?

This one looks interesting:

modules/cgred/init.pp

# Establishing cgroups and enforcing them using cgrules engine
#
# This module is oriented towards the workflow of defining
# cgroups and settings by logical unit and using the
# cgrulesengd to enforce.  The init script for cgrulesengd
# applies cgroups to this effect.
#

class cgred {
    package { [
        'cgroup-bin',
...

In T212824#4922929, @Dzahn wrote:

In T212824#4922713, @elukey wrote:

@aborrero has already done a similar thing for the tool-forge hosts. Anything that we can share with puppet?

This one looks interesting:

modules/cgred/init.pp

cgred is deprecated. You can find what I did for Toolforge here: T210098: Port cgroup restrictions and definitions to systemd/stretch for bastions, specifically T210098#4773756.

TL;DR: I just created this doc page: https://wikitech.wikimedia.org/wiki/Systemd_resource_control

@aborrero thanks a lot! As far as I can see the limits are applied for each user separately, but my use case is a bit different - I'd need to add a user login session (and hence all its processes) into a cgroup, but only if the user belongs to certain groups. Basically I'd like to allow a playground for people to crunch data as they need, but in a way that doesn't saturate all the machine leading to OOMs. Do you think that it is feasible?

EDIT: I am reading about user-.slice, missed at first pass, will see how it work first!

Change 488077 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce systemd::slice::user

https://gerrit.wikimedia.org/r/488077

Change 488078 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce profile::analytics::cluster::limits::statistics

https://gerrit.wikimedia.org/r/488078

Change 488077 merged by Elukey:
[operations/puppet@production] Introduce systemd::slice::all_users

https://gerrit.wikimedia.org/r/488077

Mentioned in SAL (#wikimedia-operations) [2019-02-12T17:54:12Z] <chaomodus> notebook1003 - restarted nagios-nrpe-server T212824

Change 488078 abandoned by Elukey:
Introduce profile::analytics::cluster::limits::statistics

https://gerrit.wikimedia.org/r/488078

Today I checked notebook1003 using the command systemd-cgls memory, that should show how the cgroups for memory settings are related to each other. Some notes:

• notebook and spark processes are under the system.slice, and they represent the bulk of the memory consumption of the hosts
• user-.slice is not mentioned (but it is if I run systemd-cgls, without memory). IIUC it doesn't represent any cgroup but only a template.

My idea would be the following:

• create a notebook-users.slice unit that comes after system.slice, and that the notebook systemd units needs to include in their config (should be easy via puppet).
• create memory and cpu limits for notebook-users.slice, that IIUC should apply to all the processes in the slice (so a sort of global limit for notebooks).

In this way I'd have a transparent way to handle a single cgroup that limits notebook usage. Need to figure out how to make sure that spark processes follow the same path, but it could be a second step. @aborrero does it sound feasible or did I get the systemd slices wrong?

@aborrero ping :]

This is my bad, I should I have followed up on this task. There are more variables since I added my last comment to add in here, sorry for the extra ping Arturo :)

Finally it's not just me squeezing notebooks memory :)

In T212824#4967798, @elukey wrote:

Today I checked notebook1003 using the command systemd-cgls memory, that should show how the cgroups for memory settings are related to each other. Some notes:

• notebook and spark processes are under the system.slice, and they represent the bulk of the memory consumption of the hosts
• user-.slice is not mentioned (but it is if I run systemd-cgls, without memory). IIUC it doesn't represent any cgroup but only a template.

My idea would be the following:

• create a notebook-users.slice unit that comes after system.slice, and that the notebook systemd units needs to include in their config (should be easy via puppet).
• create memory and cpu limits for notebook-users.slice, that IIUC should apply to all the processes in the slice (so a sort of global limit for notebooks).

In this way I'd have a transparent way to handle a single cgroup that limits notebook usage. Need to figure out how to make sure that spark processes follow the same path, but it could be a second step. @aborrero does it sound feasible or did I get the systemd slices wrong?

It does sound feasible :-) system.slice, user.slice, etc are all default slices created by systemd. AFAIK you can create arbitrary additional systemd slices and put stuff there. I don't have any example in our current puppet tree, but shouldn't be difficult I think.

Sorry all, I overlooked this ping.

Mentioned in SAL (#wikimedia-operations) [2019-11-20T21:23:09Z] <mutante> notebook1003 - systemctl start nagios-nrpe-server (second time today already today T212824)

Change 553122 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce systemd cgroup memory limits for stat1004

https://gerrit.wikimedia.org/r/553122

Change 553122 merged by Elukey:
[operations/puppet@production] Introduce systemd cgroup memory limits for stat1004

https://gerrit.wikimedia.org/r/553122

Change 553134 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::analytics::client::limits: set percentage char in user slice

https://gerrit.wikimedia.org/r/553134

Change 553134 merged by Elukey:
[operations/puppet@production] profile::analytics::client::limits: set percentage char in user slice

https://gerrit.wikimedia.org/r/553134

Applied the change to stat1004 and executed a little test:

a = []
while True:
    print(len(a))
    a.append(' ' * 10**6)

[16357443.859514] Task in /user.slice/user-13926.slice killed as a result of limit of /user.slice/user-13926.slice
[16357443.859518] memory: usage 29566528kB, limit 29566528kB, failcnt 79
[16357443.859519] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[16357443.859520] kmem: usage 59224kB, limit 9007199254740988kB, failcnt 0
[16357443.859520] Memory cgroup stats for /user.slice/user-13926.slice: cache:4KB rss:29507300KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:29507224KB inactive_file:0KB active_file:0KB unevictable:0KB
[16357443.859531] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[16357443.859635] [ 8236]     0  8236    24848     1308      51       3        0             0 sshd
[16357443.859637] [ 8246] 13926  8246    17867     1659      40       3        0             0 systemd
[16357443.859638] [ 8247] 13926  8247    61586      604      54       4        0             0 (sd-pam)
[16357443.859641] [ 8270] 13926  8270    24848      725      50       3        0             0 sshd
[16357443.859643] [ 8271] 13926  8271     5390     1210      15       3        0             0 bash
[16357443.859644] [ 8608] 13926  8608  7381963  7376996   14422      31        0             0 python3
[16357443.859646] Memory cgroup out of memory: Kill process 8608 (python3) score 999 or sacrifice child
[16357443.869857] Killed process 8608 (python3) total-vm:29527852kB, anon-rss:29503028kB, file-rss:4956kB, shmem-rss:0kB

Change 553142 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] systemd::slice::all_users: add Debian Buster support

https://gerrit.wikimedia.org/r/553142

I think we are on a good track, next steps:

1. add support for Buster (for stat1005) - needs Cloud team's review/approval: https://gerrit.wikimedia.org/r/553142
2. wait a day to see if anything comes up with stat1004, and then apply the limits to all the stat/notebooks.

@elukey: We should apply the same treatment for stat1007 :)

Change 553244 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply profile::analytics::client::limits to stat100[6,7] and notebooks

https://gerrit.wikimedia.org/r/553244

Change 553244 merged by Elukey:
[operations/puppet@production] Apply profile::analytics::client::limits to stat100[6,7] and notebooks

https://gerrit.wikimedia.org/r/553244

In T212824#5695358, @JAllemandou wrote:

@elukey: We should apply the same treatment for stat1007 :)

Done! Only stat1005 is left, but it needs another change since it is a buster node :)

Change 553142 merged by Elukey:
[operations/puppet@production] systemd::slice::all_users: add Debian Buster support

https://gerrit.wikimedia.org/r/553142

Change 553288 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::statistics::explorer::gpu: add systemd user limits

https://gerrit.wikimedia.org/r/553288

Change 553288 merged by Elukey:
[operations/puppet@production] role::statistics::explorer::gpu: add systemd user limits

https://gerrit.wikimedia.org/r/553288

I am inclined to close this task, and re-open if more alarms will appear. We'll likely have to tune limits but I hope that the current ones will suffice :)