Page MenuHomePhabricator
Create Task
Maniphest T408586

☂️ OpenSearch on K8s: Ensure that our first tenant workload is ready for production ☂️
Closed, ResolvedPublic
Actions

Description

We are currently onboarding our first OpenSearch on K8s test environment in T408012 . But before the service can truly be considered production, we need to:

  • Benchmark the opensearch-ipoid cluster using the methods described in this Wikitech article T410762
  • Ensure alerts and dashboards are created properly T362114
  • Document the environment. Target audiences:
    • SRE supporting the platform T408616
    • service owners using the platform/ potential service owners T409556
  • Successfully deploy an opensearch-ipoid cluster in CODFW. - this won't be ready until this change is merged and confirmed working

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
opensearch on k8s: stop hard-coding JVM memory settingsBkingoperations/deployment-chartsmaster+0 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT362105 EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6
 ResolvedbkingT408586 ☂️ OpenSearch on K8s: Ensure that our first tenant workload is ready for production ☂️
 ResolvedbrouberolT408616 Create SRE-scoped documentation for OpenSearch on K8s
 ResolvedRKemperT362114 OpenSearch on K8s: Create Dashboards
 ResolvedbkingT408640 OpenSearch on K8s: Create Alerts
 ResolvedbkingT402833 Opensearch on K8s: test common operations
 DuplicateNoneT404908 Test standard operation on OpenSearch running on DSE k8s
 ResolvedbkingT408919 OpenSearch on K8s: Create separate user for the OpenSearch operator
 ResolvedbkingT409501 OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc
 ResolvedbkingT409556 OpenSearch on K8s: Create service owner/potential service owner-scope documentation
 ResolvedbkingT410762 OpenSearch on K8s: Benchmark cluster
 ResolvedbkingT410956 OpenSearch on K8s: Allow applications to target specific DC for writes
 ResolvedbkingT412447 OpenSearch on K8s: Monitor and rotate TLS certificates
 ResolvedbkingT414037 Consider enabling Blackbox K8s autodiscovery on dse-k8s Prometheus instance
 ResolvedbkingT416921 OpenSearch on K8s: fix cluster resource allocation in opensearch-cluster chart/values
 OpenGehelT418175 Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform
 ResolvedbkingT419289 Ensure OpenSearch on k8s clusters can safely use envoy TLS termination
 In ProgressbkingT422378 Move OpenSearch-iPoid behind TLS/Update to OpenSearch 3.x

Event Timeline

bking created this task.Oct 28 2025, 3:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2025, 3:47 PM
bking changed the task status from Open to In Progress.Oct 28 2025, 3:47 PM
bking claimed this task.
bking triaged this task as High priority.
bking added a project: Data-Platform-SRE (2025.10.17 - 2025.11.07).
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.
bking added a parent task: T362105: EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6.Oct 28 2025, 9:31 PM
bking updated the task description. (Show Details)Oct 28 2025, 9:33 PM
brouberol added a subtask: T362114: OpenSearch on K8s: Create Dashboards.Oct 30 2025, 11:38 AM
brouberol added a subtask: T408640: OpenSearch on K8s: Create Alerts.Oct 30 2025, 11:43 AM
brouberol removed a subtask: T408643: OpenSearch on K8s: Deploy an OpenSearch cluster in dse-k8s-codfw.
brouberol added a subtask: T396501: Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters.Oct 30 2025, 11:45 AM
brouberol added a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.
brouberol added a subtask: T402833: Opensearch on K8s: test common operations.
brouberol removed a subtask: T396501: Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters.Oct 30 2025, 2:21 PM
brouberol removed a subtask: T402833: Opensearch on K8s: test common operations.Oct 30 2025, 2:33 PM
brouberol removed a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.
brouberol added a subtask: T402833: Opensearch on K8s: test common operations.
brouberol added a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.
Gehel added a project: OKR-Work.Oct 31 2025, 8:31 AM
bking mentioned this in T408953: OpenSearch on K8s: disable logging to local container storage.Oct 31 2025, 8:27 PM
bking added a subtask: T409501: OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc.Nov 6 2025, 10:17 PM
bking changed the status of subtask T408919: OpenSearch on K8s: Create separate user for the OpenSearch operator from Open to In Progress.Nov 6 2025, 11:09 PM
bking changed the status of subtask T408616: Create SRE-scoped documentation for OpenSearch on K8s from Open to In Progress.
Gehel edited projects, added: Data-Platform-SRE (2025.11.07 - 2025.11.28); removed: Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:31 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
bking closed subtask T408919: OpenSearch on K8s: Create separate user for the OpenSearch operator as Resolved.Nov 7 2025, 10:00 PM
bking changed the status of subtask T409501: OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc from Open to In Progress.Nov 10 2025, 2:45 PM
bking updated the task description. (Show Details)Nov 10 2025, 3:21 PM
bking closed subtask T409501: OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc as Resolved.Nov 14 2025, 9:23 PM
gerritbot added a comment.Nov 20 2025, 3:48 PM

Change #1207893 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] opensearch on k8s: stop hard-coding JVM memory settings

https://gerrit.wikimedia.org/r/1207893

gerritbot added a project: Patch-For-Review.Nov 20 2025, 3:48 PM
gerritbot added a comment.Nov 20 2025, 3:55 PM

Change #1207893 merged by Bking:

[operations/deployment-charts@master] opensearch on k8s: stop hard-coding JVM memory settings

https://gerrit.wikimedia.org/r/1207893

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2025, 4:32 PM
bking created subtask T410762: OpenSearch on K8s: Benchmark cluster.Nov 21 2025, 5:16 PM
bking updated the task description. (Show Details)
kostajh subscribed.Nov 21 2025, 10:06 PM
Gehel added a subtask: T410956: OpenSearch on K8s: Allow applications to target specific DC for writes.Dec 2 2025, 2:37 PM
bking closed subtask T410956: OpenSearch on K8s: Allow applications to target specific DC for writes as Resolved.Dec 11 2025, 6:40 PM
bking created subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates.Dec 11 2025, 7:26 PM
bking changed the status of subtask T408640: OpenSearch on K8s: Create Alerts from Open to In Progress.Jan 5 2026, 3:20 PM
Gehel edited projects, added: Data-Platform-SRE (2026.01.05 - 2026.01.23); removed: Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:14 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.
bking closed subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates as Resolved.Jan 9 2026, 11:34 PM
bking reopened subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates as In Progress.Jan 13 2026, 2:32 PM
bking changed the status of subtask T409556: OpenSearch on K8s: Create service owner/potential service owner-scope documentation from Open to In Progress.Jan 13 2026, 3:56 PM
bking closed subtask T408640: OpenSearch on K8s: Create Alerts as Resolved.Jan 20 2026, 3:53 PM
bking closed subtask T410762: OpenSearch on K8s: Benchmark cluster as Resolved.Jan 21 2026, 6:20 PM
bking closed subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates as Resolved.Jan 21 2026, 6:23 PM
Gehel edited projects, added: Data-Platform-SRE (2026.01.23 - 2026.02.13); removed: Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:03 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.
bking closed subtask T362114: OpenSearch on K8s: Create Dashboards as Resolved.Feb 5 2026, 7:03 PM
bking added a subtask: T416714: OpenSearch on K8s: Create separate admin user for cluster operations.Feb 6 2026, 4:37 PM
bking closed subtask T402833: Opensearch on K8s: test common operations as Resolved.Feb 9 2026, 4:11 PM
bking updated the task description. (Show Details)Feb 9 2026, 7:52 PM
bking added a subtask: T416921: OpenSearch on K8s: fix cluster resource allocation in opensearch-cluster chart/values.Feb 9 2026, 7:53 PM
bking removed a subtask: T416714: OpenSearch on K8s: Create separate admin user for cluster operations.Feb 12 2026, 9:01 PM
bking closed subtask T416921: OpenSearch on K8s: fix cluster resource allocation in opensearch-cluster chart/values as Resolved.Feb 13 2026, 4:33 PM
Gehel edited projects, added: Data-Platform-SRE (2026-02-13 - 2026-03-06); removed: Data-Platform-SRE (2026.01.23 - 2026.02.13).Feb 16 2026, 3:34 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
bking added a subtask: T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform.Feb 23 2026, 8:52 PM
bking closed subtask T408616: Create SRE-scoped documentation for OpenSearch on K8s as Resolved.Feb 26 2026, 8:45 PM
bking closed subtask T409556: OpenSearch on K8s: Create service owner/potential service owner-scope documentation as Resolved.
bking updated the task description. (Show Details)Mar 2 2026, 4:19 PM
BTullis renamed this task from ☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️ to ☂️ OpenSearch on K8s: Ensure that our first tenant workload is ready for production ☂️.Mar 5 2026, 12:14 PM
BTullis updated the task description. (Show Details)
BTullis updated the task description. (Show Details)
Gehel edited projects, added: Data-Platform-SRE (2026-03-06 - 2026-03-27); removed: Data-Platform-SRE (2026-02-13 - 2026-03-06).Mar 6 2026, 2:02 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
Gehel closed this task as Resolved.Mar 9 2026, 11:44 AM
Gehel moved this task from In Progress to Done on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
Gehel subscribed.

IPoid is deployed, so this can be considered completed.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.Mar 13 2026, 8:29 AM
bking changed the status of subtask T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform from Open to Stalled.Apr 6 2026, 1:55 PM
BTullis changed the status of subtask T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform from Stalled to Open.Apr 13 2026, 1:54 PM
bking mentioned this in T424248: ☂️Migrate non-cirrus indices from production OpenSearch to OpenSearch on k8s ☂️.Apr 23 2026, 5:05 PM
BTullis closed subtask T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform as Resolved.May 22 2026, 3:40 PM
RLazarus reopened subtask T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform as Open.May 22 2026, 3:48 PM
Gehel closed subtask T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform as Resolved.May 29 2026, 7:12 AM
RLazarus reopened subtask T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform as Open.Jun 11 2026, 10:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits