Page MenuHomePhabricator

Enable CPU performance governor on Relforge, Cloudelastic, and Elasticsearch hosts
Closed, DeclinedPublic

Description

As we migrate from Elasticsearch to Opensearch in T370147, we'll be operating the cluster with a smaller number of hosts. Additionally, there could be performance regressions that are not apparent until we're running at scale in a production cluster.

Under these circumstances, we'd like to maximize the available compute power by enabling the CPU performance governor. For context, we enabled the governor for WDQS hosts in T336443 and saw a significant increase in performance.

Note that this will increase electricity usage, so we'll need to involve DC Ops before making any changes.

Creating this ticket to:

  • Provide the full scope of proposed changes in terms of chassis/rack/row. The fault tolerance tool should be helpful for this.
  • Discuss with DC Ops and come up with an agreed-upon plan.
  • Implement the plan

Event Timeline

bking updated the task description. (Show Details)
bking changed the task status from Open to In Progress.Feb 20 2025, 9:17 PM
bking claimed this task.
bking triaged this task as Medium priority.
bking added projects: ops-eqiad, ops-codfw.

Summary of email just sent to DC Ops:

I've created this Google sheet that breaks down the Elastic hosts by rack and row. Are you able to take a look and see which racks/rows have enough power capacity to enable the Performance governor?

Hit me up on IRC (inflatador) if you need more info.

Change #1122660 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: enable perf governor, remove unused host hieradata

https://gerrit.wikimedia.org/r/1122660

Change #1122660 merged by Bking:

[operations/puppet@production] elastic: enable perf governor, remove unused host hieradata

https://gerrit.wikimedia.org/r/1122660

Unfortunately, I just now remembered that the Performance governor only helps R450 or later chassis. And we only have 2 of those in production eqiad at the moment (elastic1103 and elastic1107).
Since the power increase for 2 hosts is minimal, I went ahead and enabled the governor for elastic1103 and elastic1107. I've also updated the Google Sheet so that it only contains R450 or "newer" (in our case, we have a couple of brand-new Supermicro chassis as well).

those are racked in d4 ,f5 should not have any problems with power from eqiad side @bking

elastic2089 in A4 and , elastic2083, 2102, and 2103 in C7 are the best candidates at codfw

Change #1131775 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: enable performance governor for selected hosts

https://gerrit.wikimedia.org/r/1131775

Change #1131775 merged by Bking:

[operations/puppet@production] elastic: enable performance governor for selected hosts

https://gerrit.wikimedia.org/r/1131775

Change #1190756 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: re-enable CPU performance governor

https://gerrit.wikimedia.org/r/1190756

Change #1190756 merged by Bking:

[operations/puppet@production] cirrussearch: re-enable CPU performance governor

https://gerrit.wikimedia.org/r/1190756

We don't have enough evidence that the performance governor is helping, so we've decided to revert to the default scaling governor. I've created T435400 for this purpose. Closing...