Maniphest T408586

☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️
Open, In Progress, HighPublic
Actions

Assigned To

Authored By

	bking
	Oct 28 2025, 3:47 PM

Tags

Referenced Files

None

Subscribers

Description

We are currently onboarding our first OpenSearch on K8s test environment in T408012 . But before the service can truly be considered production, we need to:

Benchmark the cluster using the methods described in this Wikitech article T410762
Ensure alerts and dashboards are created properly T362114
Document the environment. Target audiences:
- SRE supporting the platform T408616
- service owners using the platform/ potential service owners T409556
Successfully deploy an cluster in CODFW.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	opensearch on k8s: stop hard-coding JVM memory settings	operations/deployment-charts	master	+0 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T362105 EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6
		In Progress		bking	T408586 ☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️
		In Progress		bking	T408616 Create SRE-scoped documentation for OpenSearch on K8s
		In Progress		RKemper	T362114 OpenSearch on K8s: Create Dashboards
		Resolved		bking	T408640 OpenSearch on K8s: Create Alerts
		In Progress		bking	T402833 Opensearch on K8s: test common operations
		Duplicate		None	T404908 Test standard operation on OpenSearch running on DSE k8s
		Resolved		bking	T408919 OpenSearch on K8s: Create separate user for the OpenSearch operator
		Resolved		bking	T409501 OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc
		In Progress		bking	T409556 OpenSearch on K8s: Create service owner/potential service owner-scope documentation
		Resolved		bking	T410762 OpenSearch on K8s: Benchmark cluster
		Resolved		bking	T410956 OpenSearch on K8s: Allow applications to target specific DC for writes
		Resolved		bking	T412447 OpenSearch on K8s: Monitor and rotate TLS certificates
		Resolved		bking	T414037 Consider enabling Blackbox K8s autodiscovery on dse-k8s Prometheus instance

Event Timeline

bking created this task.Oct 28 2025, 3:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2025, 3:47 PM

bking changed the task status from Open to In Progress.Oct 28 2025, 3:47 PM

bking claimed this task.

bking triaged this task as High priority.

bking added a project: Data-Platform-SRE (2025.10.17 - 2025.11.07).

bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.

bking added a parent task: T362105: EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6.Oct 28 2025, 9:31 PM

bking updated the task description. (Show Details)Oct 28 2025, 9:33 PM

brouberol added a subtask: T362114: OpenSearch on K8s: Create Dashboards.Oct 30 2025, 11:38 AM

brouberol added a subtask: T408640: OpenSearch on K8s: Create Alerts.Oct 30 2025, 11:43 AM

brouberol removed a subtask: T408643: OpenSearch on K8s: Deploy an OpenSearch cluster in dse-k8s-codfw.

brouberol added a subtask: T396501: Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters.Oct 30 2025, 11:45 AM

brouberol added a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.

brouberol added a subtask: T402833: Opensearch on K8s: test common operations.

brouberol removed a subtask: T396501: Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters.Oct 30 2025, 2:21 PM

brouberol removed a subtask: T402833: Opensearch on K8s: test common operations.Oct 30 2025, 2:33 PM

brouberol removed a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.

brouberol added a subtask: T402833: Opensearch on K8s: test common operations.

brouberol added a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.

Gehel added a project: OKR-Work.Oct 31 2025, 8:31 AM

bking mentioned this in T408953: OpenSearch on K8s: disable logging to local container storage.Oct 31 2025, 8:27 PM

bking added a subtask: T409501: OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc.Nov 6 2025, 10:17 PM

bking changed the status of subtask T408919: OpenSearch on K8s: Create separate user for the OpenSearch operator from Open to In Progress.Nov 6 2025, 11:09 PM

bking changed the status of subtask T408616: Create SRE-scoped documentation for OpenSearch on K8s from Open to In Progress.

Gehel edited projects, added Data-Platform-SRE (2025.11.07 - 2025.11.28); removed Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:31 PM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.

bking closed subtask T408919: OpenSearch on K8s: Create separate user for the OpenSearch operator as Resolved.Nov 7 2025, 10:00 PM

bking changed the status of subtask T409501: OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc from Open to In Progress.Nov 10 2025, 2:45 PM

bking updated the task description. (Show Details)Nov 10 2025, 3:21 PM

bking closed subtask T409501: OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc as Resolved.Nov 14 2025, 9:23 PM

Change #1207893 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] opensearch on k8s: stop hard-coding JVM memory settings

https://gerrit.wikimedia.org/r/1207893

gerritbot added a project: Patch-For-Review.Nov 20 2025, 3:48 PM

Change #1207893 merged by Bking:

[operations/deployment-charts@master] opensearch on k8s: stop hard-coding JVM memory settings

https://gerrit.wikimedia.org/r/1207893

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2025, 4:32 PM

bking created subtask T410762: OpenSearch on K8s: Benchmark cluster.Nov 21 2025, 5:16 PM

bking updated the task description. (Show Details)

kostajh subscribed.Nov 21 2025, 10:06 PM

Gehel added a subtask: T410956: OpenSearch on K8s: Allow applications to target specific DC for writes.Dec 2 2025, 2:37 PM

bking closed subtask T410956: OpenSearch on K8s: Allow applications to target specific DC for writes as Resolved.Dec 11 2025, 6:40 PM

bking created subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates.Dec 11 2025, 7:26 PM

bking changed the status of subtask T408640: OpenSearch on K8s: Create Alerts from Open to In Progress.Mon, Jan 5, 3:20 PM

Gehel edited projects, added Data-Platform-SRE (2026.01.05 - 2026.01.23); removed Data-Platform-SRE (2025.11.07 - 2025.11.28).Thu, Jan 8, 1:14 PM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.

bking closed subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates as Resolved.Fri, Jan 9, 11:34 PM

bking reopened subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates as In Progress.Tue, Jan 13, 2:32 PM

bking changed the status of subtask T409556: OpenSearch on K8s: Create service owner/potential service owner-scope documentation from Open to In Progress.Tue, Jan 13, 3:56 PM

bking closed subtask T408640: OpenSearch on K8s: Create Alerts as Resolved.Tue, Jan 20, 3:53 PM

bking closed subtask T410762: OpenSearch on K8s: Benchmark cluster as Resolved.Wed, Jan 21, 6:20 PM

bking closed subtask T412447: OpenSearch on K8s: Monitor and rotate TLS certificates as Resolved.Wed, Jan 21, 6:23 PM

Gehel edited projects, added Data-Platform-SRE (2026.01.23 - 2026.02.13); removed Data-Platform-SRE (2026.01.05 - 2026.01.23).Fri, Jan 23, 10:03 AM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.