Page MenuHomePhabricator
Create Task
Maniphest T183146

Monitor resource usage on a per-cgroup basis
Closed, ResolvedPublic
Actions

Description

We currently monitor and plot CPU, memory and I/O usage on a per-host basis.

In order to monitor resource usage on a service-basis, we should enable cgroups accounting in systemd, either globally or on specific units.

DefaultCPUAccounting, DefaultBlockIOAccounting and DefaultMemoryAccounting can be used to enable cgroups accounting system-wide, see systemd-system.conf(5). Similar settings (CPUAccounting and friends) can be used to enable the feature on selected units. See systemd.resource-control(5).

cadvisor can be used to export all available metrics to prometheus, using id as the label to distinguish between cgroups.

container_cpu_system_seconds_total{id="/system.slice/rsyslog.service"} 0.04
[...]
container_memory_rss{id="/system.slice/ssh.service"} 40960

Details

SubjectRepoBranchLines +/-
systemd: add support for network accountingoperations/puppetproduction+14 -1
prometheus: add cadvisor jobsoperations/puppetproduction+19 -1
cache: add cadvisor exporteroperations/puppetproduction+1 -0
cache: enable cgroup accountingoperations/puppetproduction+6 -12
prometheus: add cadvisor_exporter module and profileoperations/puppetproduction+36 -0
cache: enable cgroup accounting on two esams nodesoperations/puppetproduction+6 -0
cache: enable systemd resources accounting on cp4027operations/puppetproduction+3 -0
cache: enable systemd resources accounting on cp4026operations/puppetproduction+3 -0
cache: add systemd config to profile::cache::baseoperations/puppetproduction+1 -0
systemd: allow setting global accounting optionsoperations/puppetproduction+47 -0
Customize query in gerrit

Related Objects

Event Timeline

• ema created this task.Dec 18 2017, 2:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 18 2017, 2:30 PM
• ema triaged this task as Medium priority.Dec 18 2017, 2:30 PM
fgiunchedi merged a task: T108027: Collect per-cgroup cpu/mem and other system level metrics.Mar 16 2018, 3:12 PM
fgiunchedi added subscribers: gerritbot, Matanya.
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:38 PM
fgiunchedi moved this task from Inbox to Backlog on the observability board.Dec 10 2019, 2:19 PM
• ema raised the priority of this task from Medium to High.Jan 8 2020, 4:45 PM
• ema added a subscriber: MoritzMuehlenhoff.
CDanis subscribed.Jan 8 2020, 4:47 PM
gerritbot added a comment.Jan 15 2020, 1:15 PM

Change 565006 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] systemd: allow setting global accounting options

https://gerrit.wikimedia.org/r/565006

gerritbot added a project: Patch-For-Review.Jan 15 2020, 1:15 PM
gerritbot added a comment.Jan 15 2020, 1:56 PM

Change 565019 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: enable systemd resources accounting on cp4026

https://gerrit.wikimedia.org/r/565019

gerritbot added a comment.Jan 15 2020, 2:48 PM

Change 565032 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: add systemd config to profile::cache::base

https://gerrit.wikimedia.org/r/565032

gerritbot added a comment.Jan 15 2020, 2:56 PM

Change 565006 merged by Ema:
[operations/puppet@production] systemd: allow setting global accounting options

https://gerrit.wikimedia.org/r/565006

gerritbot added a comment.Jan 15 2020, 3:03 PM

Change 565032 merged by Ema:
[operations/puppet@production] cache: add systemd config to profile::cache::base

https://gerrit.wikimedia.org/r/565032

gerritbot added a comment.Jan 15 2020, 3:13 PM

Change 565019 merged by Ema:
[operations/puppet@production] cache: enable systemd resources accounting on cp4026

https://gerrit.wikimedia.org/r/565019

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2020, 4:10 PM
gerritbot added a comment.Jan 16 2020, 9:29 AM

Change 565236 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: enable systemd resources accounting on cp4027

https://gerrit.wikimedia.org/r/565236

gerritbot added a project: Patch-For-Review.Jan 16 2020, 9:29 AM
gerritbot added a comment.Jan 16 2020, 9:33 AM

Change 565236 merged by Ema:
[operations/puppet@production] cache: enable systemd resources accounting on cp4027

https://gerrit.wikimedia.org/r/565236

• ema added a comment.Jan 16 2020, 9:58 AM

I have enabled cpu, memory, and blockio cgroups accounting on cp4026 Jan 16 09:19:59 and cp4027 Jan 16 09:37:00.

We can now observe if the change has any impact when it comes to resource utilization. So far there's no difference, see for example cp4027.

cp4027-cgroups-accounting.png (1×1 px, 224 KB)

It's going to be interesting to observe impact on busier nodes (esams).

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2020, 10:10 AM
gerritbot added a comment.Feb 17 2020, 1:47 PM

Change 572669 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: enable cgroup accounting on two esams nodes

https://gerrit.wikimedia.org/r/572669

gerritbot added a project: Patch-For-Review.Feb 17 2020, 1:47 PM
• ema added a comment.Feb 17 2020, 2:09 PM

Cadvisor is not in Buster, see the package tracker. I have tried building it on boron: it turns out that the package depends on containerd, also not in buster, which depends on several golang libraries in turn not available. The binary package currently in testing and unstable (0.35.0+ds1-4) does however work on buster out of the box, I have verified that on cp4027.

Having discussed the matter with @MoritzMuehlenhoff and @fgiunchedi on IRC we came to the conclusion that, although far from optimal, the course of action to take now is simply importing the binary into buster-wikimedia.

Stashbot added a comment.Feb 17 2020, 2:17 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-17T14:17:21Z] <ema> reprepro includedeb buster-wikimedia ~ema/cadvisor_0.35.0+ds1-4_amd64.deb T183146

gerritbot added a comment.Feb 17 2020, 2:18 PM

Change 572669 merged by Ema:
[operations/puppet@production] cache: enable cgroup accounting on two esams nodes

https://gerrit.wikimedia.org/r/572669

gerritbot added a comment.Feb 17 2020, 2:43 PM

Change 572682 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Add module to configure cadvisor

https://gerrit.wikimedia.org/r/572682

gerritbot added a comment.Feb 17 2020, 3:32 PM

Change 572693 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: add cadvisor exporter

https://gerrit.wikimedia.org/r/572693

gerritbot added a comment.Feb 19 2020, 12:16 PM

Change 573267 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: enable cgroup accounting

https://gerrit.wikimedia.org/r/573267

gerritbot added a comment.Feb 19 2020, 12:22 PM

Change 572682 merged by Ema:
[operations/puppet@production] prometheus: add cadvisor_exporter module and profile

https://gerrit.wikimedia.org/r/572682

gerritbot added a comment.Feb 19 2020, 12:22 PM

Change 573267 merged by Ema:
[operations/puppet@production] cache: enable cgroup accounting

https://gerrit.wikimedia.org/r/573267

gerritbot added a comment.Feb 19 2020, 12:36 PM

Change 573272 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: add cadvisor jobs

https://gerrit.wikimedia.org/r/573272

gerritbot added a comment.Feb 19 2020, 12:46 PM

Change 572693 merged by Ema:
[operations/puppet@production] cache: add cadvisor exporter

https://gerrit.wikimedia.org/r/572693

gerritbot added a comment.Feb 19 2020, 3:33 PM

Change 573272 merged by Ema:
[operations/puppet@production] prometheus: add cadvisor jobs

https://gerrit.wikimedia.org/r/573272

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 4:11 PM
• ema closed this task as Resolved.Feb 19 2020, 4:11 PM
• ema claimed this task.

This is now done for cache nodes, see https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?orgId=1&fullscreen&panelId=80&from=now-15m&to=now

Screenshot from 2020-02-19 17-09-22.png (1×2 px, 189 KB)

gerritbot added a comment.Mar 30 2020, 9:21 AM

Change 584553 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] systemd: add support for network accounting

https://gerrit.wikimedia.org/r/584553

gerritbot added a project: Patch-For-Review.Mar 30 2020, 9:22 AM
gerritbot added a comment.Apr 2 2020, 8:32 AM

Change 584553 merged by Ema:
[operations/puppet@production] systemd: add support for network accounting

https://gerrit.wikimedia.org/r/584553

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2020, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL