Page MenuHomePhabricator

Install new wmf-opensearch-search-plugins package/roll-restart CirrusSearch clusters
Closed, ResolvedPublic

Description

Per this Gitlab commit , the OpenSearch plugins Debian package has been updated. Creating this ticket to activate these changes in production:

  • Update the package via apt
  • Roll-restart all CirrusSearch clusters/environments
    • relforge
    • cloudelastic
    • search_codfw
    • search_eqiad
  • Verify operation (look at the output of /_cat/plugins for all nodes, for example)

Details

Other Assignee
bking

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2025-09-04T21:31:03Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply new opensearch plugins pkg - bking@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-04T21:37:04Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply new opensearch plugins pkg - bking@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-04T21:38:20Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new opensearch plugins pkg - bking@cumin1002 - T403749

bking changed the task status from Open to In Progress.Sep 4 2025, 9:53 PM
bking assigned this task to RKemper.
bking triaged this task as Medium priority.
bking updated the task description. (Show Details)
bking updated Other Assignee, added: bking.

Mentioned in SAL (#wikimedia-operations) [2025-09-04T22:06:09Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new opensearch plugins pkg - bking@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-04T22:21:48Z] <ryankemper@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-04T22:45:36Z] <ryankemper@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-04T23:12:08Z] <ryankemper@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-05T00:07:15Z] <ryankemper@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-05T01:52:56Z] <ryankemper@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749

Mentioned in SAL (#wikimedia-operations) [2025-09-05T03:19:17Z] <ryankemper@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply new opensearch plugins pkg - ryankemper@cumin1002 - T403749

SREs (myself and @RKemper) believe we have applied all changes.

I've verified that the plugins have been applied by running
curl -s $EP/_cat/plugins | grep opensearch-extra-analysis-khmer | grep -v wmf6 against all endpoints (khmer is one of the new plugins, and it should contain the string wmf6. Thus any khmer plugin missing this string is assumed to be out of date).

Moving to 'Needs Review' so Search Platform software engineers can verify...