While investigating cluster quorum issues in parent ticket, we've noticed that several master-eligibles, including all hosts in EQIAD, don't have the performance governor enabled:
powersave [bking@cumin2003] ~$ sudo cumin A:cirrussearch-codfw-masters 'cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | uniq' 15 hosts will be targeted: cirrussearch[2061,2073,2076,2080-2081,2083-2084,2086,2092-2093,2100,2106,2108,2111,2115].codfw.wmnet ===== NODE GROUP ===== (11) cirrussearch[2061,2073,2076,2080-2081,2084,2086,2092-2093,2100,2111].codfw.wmnet ----- OUTPUT for command #1: 'cat /sys/devices..._governor | uniq' ----- powersave [bking@cumin2003] ~$ sudo cumin A:cirrussearch-eqiad-masters 'cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | uniq' 15 hosts will be targeted: cirrussearch[1068,1073-1076,1081,1093-1095,1098,1100,1102,1118-1119,1122].eqiad.wmnet ----- OUTPUT for command #1: 'cat /sys/devices..._governor | uniq' ----- powersave ===== NODE GROUP ===== (3) cirrussearch[1118-1119,1122].eqiad.wmnet ----- OUTPUT for command #1: 'cat /sys/devices..._governor | uniq' ----- schedutil
Creating this ticket to:
- Enable performance governor on all master-eligible hosts
- Observe changes (if any) during the cluster restart scheduled for tomorrow (ref T324335).
Note that non-masters are also responsible for reporting their status to the cluster, so it's possible we might have to enable the performance governor on all hosts. We'll reach out to DC Ops before making a change like that, as it could impact power draw.