Change 520885 merged by Jbond:
[operations/puppet@production] facter - cpu_details: add governor and scaling_driver facts

https://gerrit.wikimedia.org/r/520885

Mentioned in SAL (#wikimedia-operations) [2019-07-09T15:22:41Z] <godog> reboot ms-be2023 with oemhp_powerreg=os - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-09T15:27:47Z] <godog> reboot ms-be2024 with oemhp_powerreg=os - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-09T16:29:28Z] <godog> reboot ms-be2025 with oemhp_powerreg=os - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-09T16:42:37Z] <godog> reboot ms-be2026 with oemhp_powerreg=os - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-09T16:54:03Z] <godog> reboot ms-be2027 with oemhp_powerreg=os - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-09T16:59:58Z] <godog> reboot ms-be2039 with oemhp_powerreg=os - T225713

For some reasons I don't seem to be able to set oemhp_powerreg on ms-be2022, I'll try rebooting

</>hpiLO-> show /system1/oemhp_power1
                                     
status=0
status_tag=COMMAND COMPLETED
Wed Jul 10 12:45:14 2019
                        


/system1/oemhp_power1
  Targets
  Properties
    oemhp_powerreg=unavailable
    iLO 4 license is required.
    oemhp_PresentPower=257 Watts
    oemhp_power_micro_ver=1.0.9
    oemhp_auto_pwr=ON (Minimum delay)
  Verbs
    cd version exit show set

Mentioned in SAL (#wikimedia-operations) [2019-07-10T12:49:27Z] <godog> reboot ms-be2022 - T225713

Ok now all codfw row D for ms-be hosts is running with powersave, will leave it like that for a little while, no adverse effects observed so far. If the trend continue I'll do all ms-be hosts in codfw.

I've set oemhp_power1 for all ms-be hosts in codfw now, will start a rolling reboot of those:

ms-be2016 ms-be2017 ms-be2018 ms-be2019 ms-be2020 ms-be2021 ms-be2028 ms-be2029 ms-be2030 ms-be2031 ms-be2032 ms-be2033 ms-be2034 ms-be2035 ms-be2036

Mentioned in SAL (#wikimedia-operations) [2019-07-11T13:40:49Z] <godog> roll restart ms-be2016 ms-be2017 ms-be2018 ms-be2019 ms-be2020 ms-be2021 ms-be2028 ms-be2029 ms-be2030 ms-be2031 ms-be2032 ms-be2033 ms-be2034 ms-be2035 ms-be2036 - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-11T14:55:45Z] <gehel> setting CPU governor to performance for elastic1052 - T225713

In T225713#5324717, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2019-07-11T14:55:45Z] <gehel> setting CPU governor to performance for elastic1052 - T225713

this is just setting the governor to performance via /sys/... Once testing show this works well, I'll go through the full operation (bios + restart).

I observe a pretty significant drop in CPU usage on elastic1052 (>50% to ~25%), so that looks good. I'll wait until Monday to apply to the whole cluster.

Mentioned in SAL (#wikimedia-operations) [2019-07-11T15:28:35Z] <gehel> setting CPU governor to performance for wdqs1004 - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-12T18:49:43Z] <gehel> setting CPU governor to performance for wdqs1010 - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-15T08:22:34Z] <godog> set oemhp_powerreg=os on ms-be10[16-39] - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-15T08:48:50Z] <gehel> set oemhp_powerreg=os + reboot for elastic1054 - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-15T08:49:42Z] <gehel> correction: set oemhp_powerreg=os + reboot for elastic1052 (NOT elastic1054) - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-15T12:55:05Z] <gehel> shutting down tilerator on maps eqiad to free some CPU - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-15T12:59:19Z] <gehel> re-enabling kartotherian codfw - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-15T12:59:48Z] <gehel> depooling kartotherian eqiad - T225713

Have you planned the cloudvirt yet? I guess that is a bit more challenging since instances would have to be moved ahead of time, but I am genuinely interested in seeing whether that improves the bad CPU experience I have noticed.

Oops, the 3 logs above about maps shoudl have been on T218097

Mentioned in SAL (#wikimedia-operations) [2019-07-15T16:58:20Z] <jynus> setting labsdb1009/10/11 to performance scaling_governor T225713

FYI, after applying the above change, I expected a huge shift on reported load (even if performance didn't change) or on temperatures, given this (wikireplicas on labs) are our busiest databases on cpu resources due to long-running queries, however, I didn't see much difference, unlike other reporters, except on the temperatures of labsdb1011, none on the load or the temperatures of the others. Maybe CPU was already a problem in scaling for database load or something else? https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=1563087571551&to=1563260371552&var-datasource=eqiad%20prometheus%2Fops&var-cluster=mysql&var-instance=labsdb1009&var-instance=labsdb1010&var-instance=labsdb1011

Mentioned in SAL (#wikimedia-operations) [2019-07-16T18:02:51Z] <andrewbogott> rebooting cloudcontrol2003-dev, cloudweb2001-dev, cloudcontrol1004 for T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-17T10:30:55Z] <godog> start rolling reboot of ms-be eqiad hosts - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-18T09:09:08Z] <godog> resume swift ms-be rolling restarts - T225713

Mentioned in SAL (#wikimedia-operations) [2019-07-18T10:29:12Z] <godog> reboot wezen.codfw.wmnet - T225713

In T225713#5335975, @jcrespo wrote:

FYI, after applying the above change, I expected a huge shift on reported load (even if performance didn't change) or on temperatures, given this (wikireplicas on labs) are our busiest databases on cpu resources due to long-running queries, however, I didn't see much difference, unlike other reporters, except on the temperatures of labsdb1011, none on the load or the temperatures of the others. Maybe CPU was already a problem in scaling for database load or something else? https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=1563087571551&to=1563260371552&var-datasource=eqiad%20prometheus%2Fops&var-cluster=mysql&var-instance=labsdb1009&var-instance=labsdb1010&var-instance=labsdb1011

It is indeed possible that the CPU was already very utilized, I can see a small decrease in system CPU % but other than that things seems unchanged. I'm curious to know/see what powersave will do at the next reboot!

Mentioned in SAL (#wikimedia-operations) [2019-07-26T13:41:43Z] <jeh> updated labstore100[67].wikimedia.org performance scaling_governor T225713

In T225713#5335975, @jcrespo wrote:

FYI, after applying the above change, I expected a huge shift on reported load (even if performance didn't change) or on temperatures, given this (wikireplicas on labs) are our busiest databases on cpu resources due to long-running queries, however, I didn't see much difference, unlike other reporters, except on the temperatures of labsdb1011, none on the load or the temperatures of the others. Maybe CPU was already a problem in scaling for database load or something else? https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=1563087571551&to=1563260371552&var-datasource=eqiad%20prometheus%2Fops&var-cluster=mysql&var-instance=labsdb1009&var-instance=labsdb1010&var-instance=labsdb1011

I did the same test during the offsite in Dublin with labsdb1009, and also didn't see any major changes.

elastic[1032-1052].eqiad.wmnet,elastic[2025-2036].codfw.wmnet have been configured with set /system1/oemhp_power1 oemhp_powerreg=os. This will take effect after next rolling restart.

+ cloud-services-team for the hosts: cloudvirt[1001-1009,1012-1013,1019-1020].eqiad.wmnet

cloudvirt1014 has already been updated and cloudvirt1013 has the same CPU. From T223971, testing with a busy loop: time $(i=1; while (( i < 2000000 )); do (( i ++ )); done):

If this task audit hold true, when doing the change to cloudvirt1013 it should be faster as a result and the setting should be applied on all the other affected cloudvirt hosts.

@Andrew @bd808 @aborrero can you look at updating the bios setting for some of the affected cloudvirt?

Based on the benchmark in my previous comment, that seems to dramatically improve the CPU performance (12 seconds down to 7 seconds?!)

tungsten has been decom'ed today. removed from the list

I was doing comparative benchmarks of eqiad and codfw. @ori suggested that I look at CPU scaling as a possible reason for the discrepancy. The performance impact of setting scaling_governor to performance is indeed significant. The median service time for a page view served by mw2377 dropped from 177ms to 126ms, measured by ab with a concurrency of 1.

I looked at power usage with ipmi-oem dell get-instantaneous-power-consumption-data, every 10 seconds while 1000 requests were served. After an initial ramp up, power usage averaged 97W in powersave mode, and 127W in performance mode. The time to serve 1000 requests was 178s in powersave mode and 127s in performance mode.

The idle power consumption was similar.

powersave is also used in eqiad. It seems to me that for any kind of bursty latency-sensitive workload, powersave is an inappropriate choice.

In T225713#8130251, @tstarling wrote:

The performance impact of setting scaling_governor to performance is indeed significant. The median service time for a page view served by mw2377 dropped from 177ms to 126ms, measured by ab with a concurrency of 1.

That's very interesting! The effect that you're seeing with ab with a concurrency of 1 on a depooled appserver could be the result of the hardware boosting individual cores to a turbo frequency that exceeds the level the processor could sustain on multiple cores or over a prolonged period of time, so it may not be very predictive of the kind of gains we'd see in production, but definitely worth investigating further.

Can we segment performance metrics by appserver? The simplest thing might be to turn this on on 5% of machines and see what it does to power and performance.