Page MenuHomePhabricator
Create Task
Maniphest T362105

EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6
Open, HighPublic
Actions

Description

Creating this epic as a parent task for OpenSearch on K8s (formerly known as Mutualized OpenSearch)

AC:

  • OpenSearch is running on dse-k8s-eqiad cluster
  • At least one tenant is running OpenSearch on k8s

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Introduce opensearch-operator-crds chart (1/3)operations/deployment-chartsmaster+7 K -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT362105 EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6
 DeclinedNoneT362107 eqiad: 3x VM request for new opensearch cluster
 DeclinedNoneT362106 codfw: 3x VM request for new opensearch cluster
 ResolvedbkingT360724 Gather requirements for shared opensearch cluster
 DeclinedNoneT362162 Mutualized Opensearch: create/configure/test S3 snapshot provider for Opensearch
 DeclinedNoneT362410 Mutualized Opensearch: Design scheduled snapshots
 InvalidNoneT363910 Port WMF Elastic plugins to opensearch
 Resolved• StevemuneneT396478 EPIC: Build dse-k8s-codfw Kubernetes cluster
 ResolvedBTullisT396479 Decide initial hardware footprint of dse-k8s-codfw cluster
 ResolvedBTullisT353789 Re-purpose kafka-stretch200[1-2] as DSE workers in codfw
 Resolved• StevemuneneT397293 Create dse-k8s-etcd cluster in codfw
 Resolved• StevemuneneT400037 Determine dse-k8s-codfw Kubernetes IP ranges
 ResolvedBTullisT397295 Create the dse-k8s-ctrl servers in codfw
 Resolved• StevemuneneT397297 Create helmfile.d/admin_ng structure required to bootstrap the dse-k8s-codfw cluster
 Resolved• StevemuneneT397298 Configure networking for the dse-k8s-codfw cluster
 Resolved• StevemuneneT397301 Bootstrap the dse-k8s-codfw cluster
 ResolvedbrouberolT404068 Use a different set of namespaces for deployment to each of the two the dse-k8s clusters
 Resolved• StevemuneneT404433 Deploy an echoserver service on dse-k8s-codfw behind ingress
 ResolvedBTullisT404576 Enable the Container Storage Interface (CSI) and the Ceph CSI plugin on dse-k8s-codfw cluster
 ResolvedBTullisT374923 Bring cephosd200[1-3] into service as a new cluster in codfw
 ResolvedbkingT397246 Deploy OpenSearch in dse-k8s
 ResolvedbkingT399857 Review the upstream opensearch-operator chart to determine whether is is suitable for us to use
 ResolvedbrouberolT400898 Create an opensearch-test namespace and associated kubeconfigs to experiment in
 ResolvedBTullisT399208 OpenSearch on K8s: Evaluate/Decide on TLS config
 ResolvedbkingT404906 Deploy opensearch operator on DSE k8s
 ResolvedbkingT404874 Allow the opensearch-operator to watch multiple namespaces
 ResolvedbkingT404907 Deploy a test opensearch cluster on DSE k8s, using opensearch operator
 DeclinedNoneT405985 OpenSearch on K8s: migrate from 2.7.0 to 2.8.0 or later version of the chart
 ResolvedbkingT407123 Mirror OpenSearch repos from upstream
 ResolvedbkingT407126 OpenSearch on K8s: build OpenSearch images for latest versions 2x and 3x
 ResolvedBTullisT407199 Pin opensearch and logstash related package versions in puppet to avoid updates when we mirror the upstream repositories
 ResolvedcolewhiteT410795 Logstash: No longer able to install OpenSearch 2.7.0 from internal apt repo
 ResolvedbkingT407125 Refactor our OpenSearch chart for upstream version 2.8.0
 ResolvedbkingT406876 Ensure external access to opensearch-test cluster
 ResolvedbkingT403534 Add opensearch-ipoid namespace to dse-k8s Kubernetes clusters
 ResolvedbkingT400295 OpenSearch on K8s: build a new opensearch-operator image
 ResolvedtaaviT393173 Publish Wikimedia trixie base Docker image
 ResolvedMoritzMuehlenhoffT391083 Prepare our custom installer and the base layer for Trixie
 ResolvedMoritzMuehlenhoffT392790 Use a forward port of Puppet 7 on Trixie hosts
 ResolvedbkingT402926 OpenSearch on K8s: implement vm.max_map_count sysctls and any other "important settings"
DuplicatebkingT403300 OpenSearch on K8s: enable prometheus metrics
 ResolvedbkingT403522 Enable access to Airflow for kostajh/iPod OpenSearch project
 In ProgressbkingT408586 ☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️
 In ProgressbkingT408616 Create SRE-scoped documentation for OpenSearch on K8s
 ResolvedRKemperT362114 OpenSearch on K8s: Create Dashboards
 ResolvedbkingT408640 OpenSearch on K8s: Create Alerts
 In ProgressbkingT402833 Opensearch on K8s: test common operations
 DuplicateNoneT404908 Test standard operation on OpenSearch running on DSE k8s
 ResolvedbkingT408919 OpenSearch on K8s: Create separate user for the OpenSearch operator
 ResolvedbkingT409501 OpenSearch on K8s: align K8s resources in Helm chart with what was promised in design doc
 In ProgressbkingT409556 OpenSearch on K8s: Create service owner/potential service owner-scope documentation
 ResolvedbkingT410762 OpenSearch on K8s: Benchmark cluster
 ResolvedbkingT410956 OpenSearch on K8s: Allow applications to target specific DC for writes
 ResolvedbkingT412447 OpenSearch on K8s: Monitor and rotate TLS certificates
 ResolvedbkingT414037 Consider enabling Blackbox K8s autodiscovery on dse-k8s Prometheus instance
 OpenNoneT416714 OpenSearch on K8s: Create separate admin user for cluster operations
 ResolvedbkingT408643 OpenSearch on K8s: Deploy an OpenSearch cluster in dse-k8s-codfw
 Resolved• StevemuneneT408779 Create an opensearch namespace for codfw
 OpenNoneT396501 Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters
 OpenbkingT416167 Create SLOs for OpenSearch on k8s
 OpenbkingT416269 Create SLIs for opensearch on k8s

Event Timeline

bking created this task.Apr 8 2024, 7:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2024, 7:23 PM
bking updated the task description. (Show Details)Apr 8 2024, 7:24 PM
bking updated the task description. (Show Details)
bking added a subtask: T362107: eqiad: 3x VM request for new opensearch cluster.Apr 8 2024, 7:33 PM
bking added a subtask: T362106: codfw: 3x VM request for new opensearch cluster.
bking added a subtask: T360724: Gather requirements for shared opensearch cluster.Apr 8 2024, 8:25 PM
Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Apr 15 2024, 1:04 PM
Gehel triaged this task as Medium priority.Apr 15 2024, 1:05 PM
Gehel moved this task from Scratch to Epics on the Data-Platform-SRE board.Apr 15 2024, 1:07 PM
bking closed subtask T360724: Gather requirements for shared opensearch cluster as Resolved.May 3 2024, 3:32 PM
Gehel added a project: Epic.May 7 2024, 8:46 AM
bking added subscribers: BTullis, Gehel, • Stevemunene and 2 others.May 13 2024, 3:39 PM
kostajh subscribed.May 13 2024, 3:56 PM
kostajh mentioned this in T357753: Build next iteration of IPoid using OpenSearch/ElasticSearch as backend.May 20 2024, 7:26 PM
Gehel moved this task from Epics to Quarterly Goals on the Data-Platform-SRE board.Jul 9 2024, 12:49 PM
brennen mentioned this in T373130: Revisit Elasticsearch in Phorge.Aug 22 2024, 5:01 PM
Gehel added a comment.Sep 6 2024, 12:39 PM

Given the recent licensing change of Elasticsearch to AGPL, we are probably going to use Elasticsearch instead of OpenSearch. The concepts stay the same, the technical stack is slightly different.

BTullis mentioned this in T373630: Investigate recent rise in Unique Devices.Oct 1 2024, 1:38 PM
• XiaoXiao-WMF subscribed.Dec 13 2024, 3:35 PM
Gehel moved this task from Quarterly Goals to Infrastructure on the Data-Platform-SRE board.Feb 11 2025, 2:08 PM
bking closed subtask T363910: Port WMF Elastic plugins to opensearch as Invalid.May 9 2025, 2:15 PM
• TheGoldenEagle added a mock: M332: MediaWiki Skin Variables and Codex Tokens Architecture.May 11 2025, 9:51 PM
• TheGoldenEagle subscribed.
Aklapper removed a mock: M332: MediaWiki Skin Variables and Codex Tokens Architecture.May 12 2025, 9:58 AM
freephile subscribed.May 16 2025, 8:40 PM
BTullis added a subtask: T374923: Bring cephosd200[1-3] into service as a new cluster in codfw.Jun 10 2025, 3:11 PM
Gehel closed subtask T362107: eqiad: 3x VM request for new opensearch cluster as Declined.Jun 17 2025, 1:33 PM
Gehel closed subtask T362106: codfw: 3x VM request for new opensearch cluster as Declined.
BTullis mentioned this in T374923: Bring cephosd200[1-3] into service as a new cluster in codfw.Jun 27 2025, 1:56 PM
bking closed subtask T362162: Mutualized Opensearch: create/configure/test S3 snapshot provider for Opensearch as Declined.Jul 7 2025, 2:48 PM
bking added a subtask: T392222: Create ops-focused OpenSearch dashboard.Jul 7 2025, 10:20 PM
BTullis closed subtask T374923: Bring cephosd200[1-3] into service as a new cluster in codfw as Resolved.Jul 9 2025, 6:16 PM
bking renamed this task from EPIC: Mutualized opensearch cluster to EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster).Jul 11 2025, 6:51 PM
bking updated the task description. (Show Details)
bking mentioned this in T399335: OpenSearch on K8s: build a new OpenSearch 2 Docker image.
bking mentioned this in Unknown Object (Task).Jul 16 2025, 9:47 PM
bking mentioned this in Unknown Object (Task).Jul 16 2025, 9:58 PM
gerritbot added a comment.Jul 29 2025, 2:20 PM

Change #1173947 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] Introduce opensearch-operator-crds chart

https://gerrit.wikimedia.org/r/1173947

gerritbot added a project: Patch-For-Review.Jul 29 2025, 2:20 PM
brouberol changed the status of subtask T397246: Deploy OpenSearch in dse-k8s from Open to In Progress.Jul 31 2025, 1:06 PM
Ottomata subscribed.Aug 7 2025, 1:56 PM
bking added a subtask: T400295: OpenSearch on K8s: build a new opensearch-operator image.Aug 11 2025, 7:05 PM
gerritbot added a comment.Aug 25 2025, 10:11 PM

Change #1181797 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] [WIP]:dse-k8s-worker: Add sysctl setting that's required for OpenSearch

https://gerrit.wikimedia.org/r/1181797

bking changed the status of subtask T402926: OpenSearch on K8s: implement vm.max_map_count sysctls and any other "important settings" from Open to In Progress.Aug 26 2025, 1:49 PM
bking changed the status of subtask T402833: Opensearch on K8s: test common operations from Open to In Progress.Aug 26 2025, 2:13 PM
bking changed the status of subtask T400295: OpenSearch on K8s: build a new opensearch-operator image from Open to In Progress.Aug 26 2025, 2:24 PM
bking closed subtask T400295: OpenSearch on K8s: build a new opensearch-operator image as Resolved.Aug 27 2025, 2:26 PM
bking mentioned this in T403301: Discuss OpenSearch 3 roadmap/future improvements.Aug 29 2025, 8:39 PM
gerritbot added a comment.Sep 3 2025, 5:02 PM

Change #1173947 merged by jenkins-bot:

[operations/deployment-charts@master] Introduce opensearch-operator-crds chart (1/3)

https://gerrit.wikimedia.org/r/1173947

bking closed subtask T402926: OpenSearch on K8s: implement vm.max_map_count sysctls and any other "important settings" as Resolved.Sep 4 2025, 4:28 PM
Gehel renamed this task from EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) to EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - WE4.2.6.Sep 5 2025, 1:13 PM
bking changed the status of subtask T403300: OpenSearch on K8s: enable prometheus metrics from Open to In Progress.Sep 9 2025, 2:27 PM
Gehel added a subtask: T403534: Add opensearch-ipoid namespace to dse-k8s Kubernetes clusters.Sep 12 2025, 3:12 PM
brouberol changed the status of subtask T404874: Allow the opensearch-operator to watch multiple namespaces from Open to In Progress.Sep 17 2025, 3:07 PM
Gehel removed a subtask: T392222: Create ops-focused OpenSearch dashboard.Sep 17 2025, 7:31 PM
Gehel removed a subtask: T404874: Allow the opensearch-operator to watch multiple namespaces.
bking removed a subtask: T403534: Add opensearch-ipoid namespace to dse-k8s Kubernetes clusters.Sep 17 2025, 7:47 PM
BTullis raised the priority of this task from Medium to High.Sep 18 2025, 1:02 AM
Gehel renamed this task from EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - WE4.2.6 to EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6.Sep 26 2025, 9:41 AM
BTullis closed subtask T396478: EPIC: Build dse-k8s-codfw Kubernetes cluster as Resolved.Oct 13 2025, 10:32 PM
bking added a subtask: T403522: Enable access to Airflow for kostajh/iPod OpenSearch project.Oct 14 2025, 4:02 PM
bking closed subtask T397246: Deploy OpenSearch in dse-k8s as Resolved.Oct 22 2025, 6:39 PM
bking changed the status of subtask T403522: Enable access to Airflow for kostajh/iPod OpenSearch project from Open to In Progress.Oct 28 2025, 1:42 PM
bking changed the status of subtask T362114: OpenSearch on K8s: Create Dashboards from Open to In Progress.Oct 28 2025, 3:49 PM
bking added a subtask: T408586: ☂️ OpenSearch on K8s: Ensure platform is ready for production ☂️.Oct 28 2025, 9:31 PM
brouberol removed a subtask: T362114: OpenSearch on K8s: Create Dashboards.Oct 30 2025, 11:38 AM
brouberol added a subtask: T408643: OpenSearch on K8s: Deploy an OpenSearch cluster in dse-k8s-codfw.Oct 30 2025, 11:44 AM
brouberol added a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.
brouberol removed a subtask: T396501: Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters.
brouberol removed a subtask: T404908: Test standard operation on OpenSearch running on DSE k8s.
brouberol removed a subtask: T402833: Opensearch on K8s: test common operations.
brouberol added a subtask: T396501: Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters.Oct 30 2025, 2:21 PM
bking changed the status of subtask T408643: OpenSearch on K8s: Deploy an OpenSearch cluster in dse-k8s-codfw from Open to In Progress.Nov 18 2025, 7:44 PM
Gehel mentioned this in T409898: Set up OpenSearch instance supporting vector search.Nov 19 2025, 10:33 AM
Gehel moved this task from Infrastructure to Quarterly Goals on the Data-Platform-SRE board.Nov 20 2025, 2:16 PM
bking closed subtask T408643: OpenSearch on K8s: Deploy an OpenSearch cluster in dse-k8s-codfw as Resolved.Mon, Jan 12, 2:22 PM
bking mentioned this in T414345: Consider donating our time to enhance the official OpenSearch Prometheus exporter.Mon, Jan 12, 2:48 PM
brouberol closed subtask T403522: Enable access to Airflow for kostajh/iPod OpenSearch project as Resolved.Tue, Jan 27, 9:01 PM
Gehel created subtask T416167: Create SLOs for OpenSearch on k8s.Mon, Feb 2, 1:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits