Page MenuHomePhabricator
Create Task
Maniphest T383811

Ensure Search Platform-owned Elasticsearch cookbooks can handle Opensearch
Open, In Progress, HighPublic
Actions

Description

We need to ensure our cookbooks can still handle common operations (banning nodes, restarting services, etc) during and after our migration from Elastic -> OpenSearch. As noted in T391151, our current Elastic/OpenSearch cookbooks don't always meet our needs. The elasticsearch-python library dependencies in Spicerack can also affect other teams, see T390860 and this CR for examples.

Creating this ticket to:

  • Examine existing cookbooks
    • ban.py
    • force-shard-allocation.py
    • force-unfreeze.py
    • reset-read-only.py
    • restart-nginx.py
    • rolling-operation.py
  • Identify potential improvements (use other teams' OpenSearch cookbooks? Use python requests instead of ES/OS python libraries? Move the logic out of Spicerack completely?)
  • Implement the improvements

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
bking updated the task description. (Show Details)Mar 17 2025, 10:47 PM
bking updated Other Assignee, added: RKemper.
bking mentioned this in T389119: Upgrade wmf_opensearch_search_plugins .deb and restart opensearch.Mar 17 2025, 10:49 PM
bking added a subscriber: Volans.Mar 17 2025, 11:22 PM
bking updated the task description. (Show Details)Mar 18 2025, 1:19 PM
bking updated the task description. (Show Details)Mar 18 2025, 6:59 PM
bking added a comment.Mar 18 2025, 7:41 PM

Per today's conversation with my teammates in Data Platform SRE, we have decided that the rolling-operation cookbook provides enough value in terms of orchestrating the batching and healthchecks that it should be used to carry out the migration.

As such, I'm setting this task as a blocker to the production migration ( T388610 ).

bking added a parent task: T388610: Migrate production Elastic clusters to Opensearch.Mar 18 2025, 7:42 PM
gerritbot added a comment.Mar 19 2025, 9:01 PM

Change #1129373 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] rolling-operation.py: Put back a reference to nodes.start_elasticsearch()

https://gerrit.wikimedia.org/r/1129373

gerritbot added a project: Patch-For-Review.Mar 19 2025, 9:01 PM
gerritbot added a comment.Mar 19 2025, 9:50 PM

Change #1129373 merged by Ryan Kemper:

[operations/cookbooks@master] rolling-operation.py: Put back a reference to nodes.start_elasticsearch()

https://gerrit.wikimedia.org/r/1129373

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2025, 10:31 PM
bking added a comment.Mar 19 2025, 10:44 PM

Per the above patch, rolling-operation.py works...however, I did have a timeout when I used the --allow-yellow flag: Error while waiting for yellow with no initializing or relocating shards . The cluster was in green, which makes me wonder if the cookbook is checking for ' yellow and no relocating shards'. We want 'green' (unqualified) OR 'yellow and no relocating shards.', so that 'green and no relocating shards' is also a valid status. Will follow up on this hunch tomorrow.

ops-monitoring-bot added a comment.Mar 20 2025, 3:20 PM

Icinga downtime and Alertmanager silence (ID=b3f79c36-c233-46b5-bfaf-6d5a09ed70df) set by bking@cumin2002 for 4:00:00 on 5 host(s) and their services with reason: troubleshooting red status

cloudelastic[1007,1009-1012].eqiad.wmnet
bking added a comment.Mar 20 2025, 3:32 PM

I ran the cookbook without the allow-yellow flag: sudo cookbook sre.elasticsearch.rolling-operation cloudelastic "try rolling operation without allow-yellow flag" --restart --task-id T389119 --nodes-per-run 1

and it still caused a red status on Cloudelastic. Until we can figure out what's going wrong, we need to either:

  • migrate without the rolling-operation cookbook, as originally planned in T388610
  • hold up the migration until we can fix the issue
Gehel edited projects, added Data-Platform-SRE (2025.03.22 - 2025.04.11); removed Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:57 AM
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Mar 21 2025, 10:16 PM
bking added a subscriber: RKemper.EditedMar 21 2025, 10:22 PM

Adding a few notes from pairing with @RKemper today:

  • We repeated the rolling operation today with the same arguments as above, and got the exact same results (red status).
  • The operation seems to work OK for the first 2 hosts, then all clusters go red when it gets to the third host.
  • I sent the command 'for n in $(cat /etc/opensearch/instances); do systemctl start $n; done' to all hosts when we were in red status, but I don't think it actually helped anything. I'm pretty sure it recovered on its own.
  • We noticed that there are unit files for both opensearch_1@cloudelastic-eqiad.service and opensearch_1@cloudelastic-chi-eqiad.service. . The unit opensearch_1@cloudelastic-eqiad.service was showing in failed state. I don't think we want both of these. red herring, you can trigger this behavior by passing systemctl start any-invalid-unit-file-name.service, which will fail and set off alerts. So I probably created this problem by trying to systemctl start opensearch_1@cloudelastic-eqiad.service (which doesn't actually have a unit file).

Further investigation is necessary.

gerritbot added a comment.Mar 24 2025, 2:45 PM

Change #1130624 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: DO NOT MERGE (just for PCC)

https://gerrit.wikimedia.org/r/1130624

gerritbot added a project: Patch-For-Review.Mar 24 2025, 2:45 PM
gerritbot added a comment.Mar 24 2025, 5:51 PM

Change #1130624 abandoned by Bking:

[operations/puppet@production] cloudelastic: DO NOT MERGE (just for PCC)

Reason:

figured out what I needed from this. closing...

https://gerrit.wikimedia.org/r/1130624

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2025, 6:30 PM
gerritbot added a comment.Mar 24 2025, 7:15 PM

Change #1130701 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: replace failed master-eligible host

https://gerrit.wikimedia.org/r/1130701

gerritbot added a project: Patch-For-Review.Mar 24 2025, 7:15 PM
gerritbot added a comment.Mar 24 2025, 7:18 PM

Change #1130701 merged by Ryan Kemper:

[operations/puppet@production] cloudelastic: replace failed master-eligible host

https://gerrit.wikimedia.org/r/1130701

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2025, 7:30 PM
gerritbot added a comment.Mar 28 2025, 8:17 PM

Change #1132024 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] relforge: enable rack awareness

https://gerrit.wikimedia.org/r/1132024

gerritbot added a project: Patch-For-Review.Mar 28 2025, 8:17 PM
gerritbot added a comment.Mar 28 2025, 8:20 PM

Change #1132024 merged by Bking:

[operations/puppet@production] relforge: enable rack awareness

https://gerrit.wikimedia.org/r/1132024

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2025, 8:31 PM
bking added a comment.Mar 31 2025, 1:21 PM

Following up, the above issue was due to cluster quorum and didn't have anything to do with the cookbook.

bking added a comment.EditedApr 3 2025, 4:27 PM

There are a few things we need to address in Spicerack before we can move forward with the migration:

  • --allow-yellow , which enables the check_yellow_w_no_moving_shards function , doesn't quite fit our scenario. We'll need a few function that allows the cluster to restart when we have UNASSIGNED ALLOCATION_FAILED replica shards. This is because OpenSearch hosts can't replicate their primary shards to Elasticsearch.

Fixed thanks to @RKemper

Lower priority/not a migration blocker:

  • --allow-yellow needs BOTH no relocating shards AND yellow status, it doesn't seem to work with no relocating shards AND green status.
gerritbot added a comment.Apr 3 2025, 4:29 PM

Change #1133967 had a related patch set uploaded (by Bking; author: Bking):

[operations/software/spicerack@master] WIP: more fine-grained shard status checks

https://gerrit.wikimedia.org/r/1133967

gerritbot added a project: Patch-For-Review.Apr 3 2025, 4:29 PM
gerritbot added a comment.Apr 3 2025, 7:57 PM

Change #1134043 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: add puppet 7 hieradata to DC-specific config

https://gerrit.wikimedia.org/r/1134043

gerritbot added a comment.Apr 3 2025, 8:07 PM

Change #1134043 merged by Bking:

[operations/puppet@production] cirrussearch: add puppet 7 hieradata to DC-specific config

https://gerrit.wikimedia.org/r/1134043

gerritbot added a comment.Apr 3 2025, 8:56 PM

Change #1133967 abandoned by Bking:

[operations/software/spicerack@master] WIP: more fine-grained shard status checks

Reason:

not needed after all

https://gerrit.wikimedia.org/r/1133967

Maintenance_bot removed a project: Patch-For-Review.Apr 3 2025, 9:30 PM
gerritbot added a comment.Apr 4 2025, 7:38 PM

Change #1131446 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] elasticsearch rolling-operation: add arguments for rename & reimage cookbooks

https://gerrit.wikimedia.org/r/1131446

gerritbot added a project: Patch-For-Review.Apr 4 2025, 7:38 PM
bking changed the status of subtask T391151: Ensure our automation can ban not-yet-existing hosts from Open to In Progress.Apr 7 2025, 2:01 PM
gerritbot added a comment.Apr 8 2025, 2:01 PM

Change #1131446 merged by Bking:

[operations/cookbooks@master] elasticsearch rolling-operation: add arguments for rename & reimage cookbooks

https://gerrit.wikimedia.org/r/1131446

gerritbot added a comment.Apr 8 2025, 10:37 PM

Change #1135133 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: handle negative caches between rename/reimage

https://gerrit.wikimedia.org/r/1135133

gerritbot added a comment.Apr 9 2025, 2:12 AM

Change #1135133 merged by Ryan Kemper:

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: handle negative caches between rename/reimage

https://gerrit.wikimedia.org/r/1135133

Maintenance_bot removed a project: Patch-For-Review.Apr 9 2025, 2:30 AM
gerritbot added a comment.Apr 10 2025, 9:22 PM

Change #1135826 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: don't use http for dhcp for reimage

https://gerrit.wikimedia.org/r/1135826

gerritbot added a project: Patch-For-Review.Apr 10 2025, 9:22 PM
Gehel edited projects, added Data-Platform-SRE (2025.04.12 - 2025.05.02); removed Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:14 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.
bking mentioned this in T390860: Elasticsearch dependency upgrade in spicerack.Apr 14 2025, 7:18 PM
gerritbot added a comment.Apr 15 2025, 4:46 PM

Change #1135826 merged by Bking:

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: use tftp, run puppet with new hostname

https://gerrit.wikimedia.org/r/1135826

Maintenance_bot removed a project: Patch-For-Review.Apr 15 2025, 5:30 PM
bking closed subtask T391151: Ensure our automation can ban not-yet-existing hosts as Resolved.Apr 15 2025, 7:14 PM
gerritbot added a comment.Apr 15 2025, 7:48 PM

Change #1136796 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: refactor external cookbook invocations

https://gerrit.wikimedia.org/r/1136796

gerritbot added a project: Patch-For-Review.Apr 15 2025, 7:48 PM
bking updated the task description. (Show Details)Apr 15 2025, 8:37 PM
bking mentioned this in T392015: Refactor/replace Search Platform-owned Elastic/OpenSearch cookbooks.
Gehel merged a task: T392015: Refactor/replace Search Platform-owned Elastic/OpenSearch cookbooks.Apr 16 2025, 7:44 AM
gerritbot added a comment.Apr 16 2025, 9:05 PM

Change #1136796 merged by Ryan Kemper:

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: refactor external cookbook invocations

https://gerrit.wikimedia.org/r/1136796

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2025, 9:31 PM
Gehel edited projects, added Data-Platform-SRE (2025.05.02 - 2025.05.23); removed Data-Platform-SRE (2025.04.12 - 2025.05.02).May 5 2025, 12:49 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
Gehel subscribed.May 7 2025, 1:50 PM

Let's finish T388610 first

Gehel moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.May 7 2025, 1:50 PM
Gehel edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE (2025.05.02 - 2025.05.23).May 23 2025, 1:07 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.
Stashbot added a comment.May 28 2025, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-28T13:19:38Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T383811 - bking@cumin2002

Stashbot added a comment.May 28 2025, 1:20 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-28T13:20:17Z] <bking@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T383811 - bking@cumin2002

gerritbot added a comment.May 28 2025, 9:17 PM

Change #1151797 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] sre.elasticsearch: remove unused cookbooks

https://gerrit.wikimedia.org/r/1151797

gerritbot added a project: Patch-For-Review.May 28 2025, 9:17 PM
Stashbot added a comment.Jun 6 2025, 5:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-06T17:06:55Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T383811 - bking@cumin2002

Stashbot added a comment.Jun 6 2025, 5:08 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-06T17:08:51Z] <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T383811 - bking@cumin2002

Stashbot added a comment.Jun 6 2025, 5:20 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-06T17:20:21Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T383811 - bking@cumin2002

Stashbot added a comment.Jun 6 2025, 7:11 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-06T19:11:22Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T383811 - bking@cumin2002

Gehel edited projects, added Data-Platform-SRE (2025.06.13 - 2025.07.04); removed Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:48 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.
BTullis edited projects, added Data-Platform-SRE (2025.07.05 - 2025.07.25); removed Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 4 2025, 4:28 PM
bking moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 8 2025, 2:52 PM

Blocking on T390860, as this changes associated with that task will affect all of the above cookbooks.

bking removed bking as the assignee of this task.Jul 8 2025, 2:52 PM
bking updated Other Assignee, removed: RKemper.
BTullis edited projects, added Data-Platform-SRE (2025.07.26 - 2025.08.15); removed Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:46 PM
BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
BTullis edited projects, added Data-Platform-SRE (2025.08.16 - 2025.09.05); removed Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:31 AM
BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
Gehel edited projects, added Data-Platform-SRE (2025.09.05 - 2025.09.26); removed Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:23 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
Gehel added a project: Essential-Work.Sep 11 2025, 12:11 PM
Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:50 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
Gehel edited projects, added Data-Platform-SRE (2025.10.17 - 2025.11.07); removed Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:48 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.
Gehel edited projects, added Data-Platform-SRE (2025.11.07 - 2025.11.28); removed Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:30 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits