Page MenuHomePhabricator

Refactor cgroups implementation/improve process observability for stat hosts
Closed, ResolvedPublic

Description

This task has 2 related goals:

  1. Unify and correct the cgroups puppet code. Currently, it is implemented in two places:

There are also minor corrections, such as setting MemoryMax (the point at which the host tries to aggressively reclaim memory) to the same value as MemoryHigh (the point at which the process is OOMkilled). The larger the gap between these two, the more the host freezes. The current difference is 40% (50% high, 90% max).

  1. Improve process observability for stat hosts

While we're currently tracking Jupyter notebook resource consumption on our stat hosts dashboard, users can also start expensive processes outside of JupyterHub. We can track these by enabling the "process option" in cadvisor (ref cadvisor docs).

Event Timeline

Change #1081987 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] statistics::explorer hosts: better visibility into processes

https://gerrit.wikimedia.org/r/1081987

I also updated the stat hosts dashboard with a panel that shows memory usage per slice .

Change #1082072 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] statistics::explorer hosts: refactor and improve cgroups implementation

https://gerrit.wikimedia.org/r/1082072

Change #1082072 merged by Bking:

[operations/puppet@production] statistics::explorer hosts: refactor and improve cgroups implementation

https://gerrit.wikimedia.org/r/1082072

Change #1081987 merged by Bking:

[operations/puppet@production] statistics::explorer hosts: better visibility into processes

https://gerrit.wikimedia.org/r/1081987

Change #1083815 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] stat hosts: guarantee minimum RAM% for system processes

https://gerrit.wikimedia.org/r/1083815

Change #1083815 merged by Bking:

[operations/puppet@production] stat hosts: guarantee minimum RAM% for system procs

https://gerrit.wikimedia.org/r/1083815

bking added a subscriber: tappof.

Thanks to @tappof , this is complete. Closing...