Per IRC conversation with @Volans , there is a Puppet resource called profile::netbox::host that "was [created] exactly to help remove all the hardcoded info for the rack/row aware services."
Creating this ticket to explore the possibility of using this for our cirrussearch hosts, instead of the disgusting regexes we're currently using .
We had a few reimage failures in CODFW due to mistakes in regex.yaml. We should experiment with this resource during the eqiad portion of the OpenSearch migration ( T388610 ). Hopefully we can avoid some of the same failures.
Change #1137272 had a related patch set uploaded (by Gehel; author: Gehel):
[operations/puppet@production] refactor(opensearch): use Netbox to get rack / row information
Tagging observability since this will impact the logging cluster. This should be a NOOP since the logging cluster does not use cluster.routing.allocation.awareness.attributes. This might be the occasion to introduce allocation awareness if this is deemed useful for this cluster.
The above CR has been approved by observability . Next, we need to think about the implementation. I was hoping we could simply replace the node row attribute in the Elastic/Opensearch REST API, but that's not an option as far as I can tell .
That means we'll need to roll-restart any cluster that changes its rack/row awareness, and as @Gehel mentioned in the above CR, this will cause some shard reshuffling as the attributes won't be exactly the same:
previously: row=A
now: row=codfw-row-a
The next step is to shut off Puppet and roll out to cloudelastic and see how it goes. If everything works out OK, we'll move to CODFW.
Change #1137272 merged by Bking:
[operations/puppet@production] refactor(opensearch): use Netbox to get rack / row information
Mentioned in SAL (#wikimedia-operations) [2025-04-30T16:59:55Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new madvise and row/rack awareness T391392 T390100 - bking@cumin2002 - T390100
Mentioned in SAL (#wikimedia-operations) [2025-04-30T17:27:51Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new madvise and row/rack awareness T391392 T390100 - bking@cumin2002 - T390100
Change #1225639 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] cirrussearch: remove defunct regexes
Change #1225639 merged by Bking:
[operations/puppet@production] cirrussearch: remove defunct regexes