We should update our kubernetes clusters to kernel 4.19 which is in Debian Stretch now (prior to going all Buster).
https://packages.debian.org/stretch/linux-image-4.19-amd64
We could start with staging ofc and if that looks okay update selected nodes in eqiad and codfw to see how they behave under higher load.
There now is a hiera key that can be set to install kernel 4.19
profile::base::linux419::enable
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | akosiaris | T244335 Upgrade kubernetes clusters to v1.16 | |||
| Resolved | JMeybohm | T262527 Update to kernel 4.19 on kubernetes nodes |
We should probably create some linux419 base profile (or similar) which
and then add that to the k8s worker role.
Change 626592 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] Add ::profile::base::linux419 to set up kernel 4.19 on stretch
Change 626592 merged by JMeybohm:
[operations/puppet@production] Add ::profile::base::linux419 to set up kernel 4.19 on stretch
Change 627867 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] Use Kernel 4.19 on kubestage1002
Change 627868 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] Use Kernel 4.19 on staging cluster nodes
Change 627867 merged by JMeybohm:
[operations/puppet@production] Use Kernel 4.19 on kubestage1002
Mentioned in SAL (#wikimedia-operations) [2020-09-17T07:55:10Z] <jayme> cordoning kubestage1002 for kernel upgrade - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-17T07:55:26Z] <jayme> draining kubestage1002 for kernel upgrade - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-17T08:25:40Z] <jayme> reboot kubestage1002 for kernel upgrade - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-17T08:43:53Z] <jayme> uncordoned kubestage1002 after kernel upgrade - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-17T08:49:48Z] <jayme> deleting some random pods in kubernetes staging to rebalance load back on kubestage1002 - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-18T07:12:08Z] <jayme> draining kubestage1001 for kernel upgrade - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-18T08:25:28Z] <jayme> reboot kubestage1001 for clean state testing - T262527
Change 627868 merged by JMeybohm:
[operations/puppet@production] Use Kernel 4.19 on staging cluster nodes
Mentioned in SAL (#wikimedia-operations) [2020-09-18T08:43:20Z] <jayme> reboot kubestage1001 for kernel upgrade - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-18T08:56:14Z] <jayme> reboot kubestage1001 for clean state - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-18T09:46:57Z] <jayme> uncordoned kubestage1001 - T262527
Mentioned in SAL (#wikimedia-operations) [2020-09-18T09:47:08Z] <jayme> deleting some random pods in kubernetes staging to rebalance load back on kubestage1001 - T262527
I've updated the second node to Kernel 4.19 as well but the throttling values don't look as good as I had hoped they would.
I ran some https://github.com/indeedeng/fibtest on empty kubestage1001 (40 core machine) on kernel 4.9 and 4.19 and they also don't show the expected difference in "CPU Usage" I would have expected. Feels like we're missing something here...
(See https://grafana.wikimedia.org/d/q-YRTAdGz/jayme-node-cfs-details?orgId=1 for a very board overview of throttling on the nodes)
| kernel | governor | fibtest threads | Iterations Completed(M) | Throttled for | CPU Usage (msecs) |
| 4.9 | performance | 1 | 1191 | 52 | 513 |
| 4.9 | performance | 10 | 1122 | 51 | 503 |
| 4.9 | performance | 20 | 960 | 52 | 506 |
| 4.9 | performance | 40 | 698 | 52 | 501 |
| 4.9 | powersave | 1 | 455 | 53 | 512 |
| 4.9 | powersave | 10 | 407 | 52 | 513 |
| 4.9 | powersave | 20 | 351 | 52 | 505 |
| 4.9 | powersave | 40 | 251 | 53 | 506 |
| kernel | governor | fibtest threads | Iterations Completed(M) | Throttled for | CPU Usage (msecs) |
| 4.19 | performance | 1 | 1252 | 52 | 516 |
| 4.19 | performance | 10 | 1063 | 51 | 504 |
| 4.19 | performance | 20 | 923 | 52 | 498 |
| 4.19 | performance | 40 | 654 | 52 | 498 |
| 4.19 | powersave | 1 | 477 | 52 | 511 |
| 4.19 | powersave | 10 | 400 | 53 | 509 |
| 4.19 | powersave | 20 | 356 | 52 | 510 |
| 4.19 | powersave | 40 | 237 | 53 | 507 |
Interestingly the majority of throttling comes from statsd_exporters (https://grafana.wikimedia.org/d/tn6gBadMz/jayme-container_cpu_cfs_throttled_seconds_total?orgId=1&var-dc=thanos&var-site=eqiad&var-prometheus=k8s-staging&var-sum_by=container_name&var-service=All) which we don't deploy with resource limits or requests (so they are all on default of 100m/100m).
When we ignore those, thinks actually look quite okay with the new kernel:
https://grafana.wikimedia.org/d/tn6gBadMz/jayme-container_cpu_cfs_throttled_seconds_total?orgId=1&var-dc=thanos&var-site=eqiad&var-prometheus=k8s-staging&var-sum_by=container_name&var-service=All&var-ignore_container_regex=%7CPOD%7C.*metrics-exporter.*
This does look pretty nice indeed. That drop there is actually quite telling. I am inclined to say we may have solved 1 problem at least.
Yeah. As just said today I will do a test with deploying resource requests and limits for the statsd exporter of eventgate-analytics to see if that changes the throttling behavior there. Then I would probably start to write the cookbook for rolling restarts of k8s clusters (T260661) in preparation of the update in eqiad and codfw (all of that in next quarter)