Page MenuHomePhabricator
Create Task
Maniphest T376653

Investigate I/O and implement cgroups on stat hosts
Closed, ResolvedPublic
Actions

Description

Creating this ticket to:

Specific tasks for stat1011:

[] Use the Prometheus metrics recorded around the time of the last stat1011 hang to estimate the upper bounds of I/O performance on stat1011 in its current configuration
[] Get hardware information on
-- stat1011's host bus adapter ( the Dell HBA355i Front, product info here )
-- stat1011's disk . This information will help us compare theoretical maximums to what we saw during the last outage.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
stat hosts: create/enable cgroups for memory and i/oBkingoperations/puppetproduction+16 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Oct 7 2024, 8:07 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 7 2024, 8:07 PM
bking changed the task status from Open to In Progress.Oct 7 2024, 8:22 PM
bking claimed this task.
bking triaged this task as Medium priority.
bking updated the task description. (Show Details)
bking updated the task description. (Show Details)
bking added a comment.Oct 7 2024, 10:19 PM

This writeup from Facebook provides an excellent real-world example of using cgroups v2 to protect workloads.

KCVelaga_WMF subscribed.Oct 8 2024, 2:17 PM
bking renamed this task from Investigate I/O and implement cgroups on stat1011 to Investigate I/O and implement cgroups on stat hosts.Oct 9 2024, 3:10 PM
bking updated the task description. (Show Details)
bking mentioned this in T376813: Implement non-cgroups-related performance optimizations on stat hosts.Oct 9 2024, 3:14 PM
gerritbot added a comment.Oct 10 2024, 1:15 PM

Change #1079281 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] stat hosts: create/enable cgroups for memory and i/o

https://gerrit.wikimedia.org/r/1079281

gerritbot added a project: Patch-For-Review.Oct 10 2024, 1:15 PM
gerritbot added a comment.Oct 10 2024, 5:13 PM

Change #1079281 merged by Bking:

[operations/puppet@production] stat hosts: create/enable cgroups for memory and i/o

https://gerrit.wikimedia.org/r/1079281

bking updated the task description. (Show Details)Oct 10 2024, 8:55 PM
bking added a comment.Oct 10 2024, 9:15 PM

Per the above PR, we have activated memory and I/O cgroups on all stat hosts. I've crossed out the rest of the AC as it's entirely possible that we'll be using Ceph homedirs instead of the current disks fairly soon (ref: this design doc). We can always take a closer look at the disks if necessary, but I'm going to close this one out for now.

bking closed this task as Resolved.Oct 10 2024, 9:16 PM
bking moved this task from Backlog - project to Done on the Data-Platform-SRE (2024.09.28 - 2024.10.18) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits