Page MenuHomePhabricator
Create Task
Maniphest T402833

Opensearch on K8s: test common operations
Closed, ResolvedPublic
Actions

Description

Per parent ticket, we need to have confidence that typical Kubernetes operations work when the OpenSearch operator is used.

Creating this ticket to work through the following operations and record results:

  • Minikube environment:
    • Change source image for OpenSearch cluster
    • Change number of replicas
    • Change resources (requests/limits)
    • Change size of PV
  • WMF Cluster
    • Change source image for OpenSearch cluster
    • Change number of replicas
    • Change resources (requests/limits)
    • Change size of PV
    • rolling restart completed
    • Drain a Kubernetes node and check that the OS cluster tolerates it nicely
    • Evict the current cluster manager and check that a new one is elected

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
opensearch-ipoid: Expand pod disk size from 30->40 GBoperations/deployment-chartsmaster+34 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Aug 25 2025, 4:50 PM
bking updated the task description. (Show Details)Aug 25 2025, 8:21 PM
bking changed the task status from Open to In Progress.Aug 26 2025, 2:12 PM
bking triaged this task as High priority.
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
bking updated the task description. (Show Details)Aug 26 2025, 4:43 PM
Gehel edited projects, added Data-Platform-SRE (2025.09.05 - 2025.09.26); removed Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:24 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
bking updated the task description. (Show Details)Sep 8 2025, 4:53 PM
Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
bking added a parent task: T397246: Deploy OpenSearch in dse-k8s.Oct 2 2025, 4:05 PM

Marking this is subtask of T397246, as we'll need to confirm all of the above operations before we consider the cluster "deployed".

bking added a comment.Oct 9 2025, 1:23 PM

Something else I noticed (which shouldn't be too surprising) is that the persistent volumes are sticking around even when I destroy the cluster. That means that when I redeploy, the new cluster won't bootstrap because it still has the data for the old cluster.

I'm working on a way to wipe out the old PVs next. It's possible we could figure out a way to bring back an existing cluster that's been accidentally destroyed, but OpenSearch on k8s is not to be used as a primary datasource for anything. In that spirit, I'm not going to work on an existing cluster recovery procedure anytime soon.

Gehel edited projects, added Data-Platform-SRE (2025.10.17 - 2025.11.07); removed Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:49 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.
Gehel added a project: OKR-Work.Oct 17 2025, 1:46 PM
brouberol removed a parent task: T397246: Deploy OpenSearch in dse-k8s.Oct 30 2025, 11:44 AM
brouberol edited parent tasks, added: T408586: ☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️; removed: T362105: EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6.
brouberol removed a parent task: T408586: ☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️.Oct 30 2025, 2:33 PM
brouberol added a parent task: T408586: ☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️.
Gehel edited projects, added Data-Platform-SRE (2025.11.07 - 2025.11.28); removed Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:31 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
bking mentioned this in T362114: OpenSearch on K8s: Create Dashboards.Nov 17 2025, 11:25 PM
Gehel updated the task description. (Show Details)Nov 18 2025, 2:50 PM
Gehel merged a task: T404908: Test standard operation on OpenSearch running on DSE k8s.
gerritbot added a comment.Jan 5 2026, 5:47 PM

Change #1223228 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] opensearch-ipoid: Expand pod disk size from 30->40 GB

https://gerrit.wikimedia.org/r/1223228

gerritbot added a project: Patch-For-Review.Jan 5 2026, 5:47 PM
gerritbot added a comment.Jan 5 2026, 7:39 PM

Change #1223228 merged by Bking:

[operations/deployment-charts@master] opensearch-ipoid: Expand pod disk size from 30->40 GB

https://gerrit.wikimedia.org/r/1223228

Maintenance_bot removed a project: Patch-For-Review.Jan 5 2026, 8:31 PM
bking updated the task description. (Show Details)Jan 6 2026, 3:01 PM
Gehel edited projects, added Data-Platform-SRE (2026.01.05 - 2026.01.23); removed Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:14 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.
Gehel edited projects, added Data-Platform-SRE (2026.01.23 - 2026.02.13); removed Data-Platform-SRE (2026.01.05 - 2026.01.23).Fri, Jan 23, 10:03 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.
bking closed this task as Resolved.Mon, Feb 9, 4:11 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

There are a couple of unfinished operations on this board:
Change number of replicas: The current design provisions exact 3 pods, so we don't really need to test this.
Drain a Kubernetes node and check that the OS cluster tolerates it nicely: We have existing workarounds for this if need be (delete a pod, storage is not deleted so downtime is minimal).

We'll repeat these same tests for the new OpenSearch operator in T414217. For the current OpenSearch operator deploy, we are finished. Closing...

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.Fri, Feb 13, 8:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits