Page MenuHomePhabricator
Create Task
Maniphest T421757

☂️ Migrate production OpenSearch clusters from 1.x-2.x ☂️
Open, Needs TriagePublic
Actions

Description

Per the OpenSearch Roadmap , OpenSearch 1.x ceased to be maintained when OpenSearch 3.x was released in May 2025.

To reduce risk and take advantage of new features, we need to migrate our production OpenSearch clusters to 2.x. Additionally, we need to migrate our hosts off Debian 11 (Bullseye) and onto Debian 13 (Trixie), this will happen in 1 step (reimage existing Debian 11/OpenSearch 1.x hosts -> Debian 13/OpenSearch 2.x hosts).

Creating this ticket to as an umbrella task for all needed work, such as:

  • Ensure Cirrussearch Puppet plans work with OpenSearch 2.x
  • Ensure Cookbooks work with OpenSearch 2.x
  • Migrate beta cluster to OpenSearch 2.x T421763
  • Migrate Cloudelastic to OpenSearch 2.x T422860
  • Drop the Cirrus Metastore T424035 (not directly related, but will make the migration easier)
  • Migrate production CODFW to OpenSearch 2.x
  • Migrate production EQIAD to OpenSearch 2.x

A tentative maintenance plan is available here

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
OpenSearch: Add required config for bootstrappingoperations/puppetproduction+27 -11
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT421757 ☂️ Migrate production OpenSearch clusters from 1.x-2.x ☂️
 In ProgressbkingT421763 Migrate beta cluster to OpenSearch 2.x
 In ProgressbkingT424100 No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep
 OpenbkingT424104 No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep
 In ProgressbkingT425585 Write lightweight OCI-image-based Puppet plans for beta cluster
 ResolvedbkingT422860 Migrate Cloudelastic to OpenSearch 2.x
 ResolvedbkingT423291 Build new wmf-opensearch-search-plugins package for opensearch 2.x/trixie and ensure we don't install/enable any unwanted plugins in prod
 ResolvedbkingT423327 Explore options for OpenSearch 2.x/3.x plugin packaging and distribution
 ResolvedbkingT423523 Handle typos/possibly update opensearch-analysis-stconvert plugin
 In ProgressdcausseT423993 Upgrade old indices in the CirrusSearch opensearch clusters
 In ProgressatsukoT424248 ☂️Migrate non-cirrus indices from production OpenSearch to OpenSearch on k8s ☂️
 StalledatsukoT425377 Migrate Ttmserver (Translatewiki application) indices from production OpenSearch to OpenSearch on k8s
 OpenNikerabbitT428168 Make Translate compatible with OpenSearch 2
 In Progressbd808T426073 Migrate toolhub indices from production OpenSearch to OpenSearch on k8s
 DuplicatebkingT424852 Investigate performance issues in cloudelastic
 In ProgressbkingT426101 Create detailed migration plan for OpenSearch 1.x-2x
 In ProgressbkingT426862 Ensure cookbooks work with OpenSearch 2.x
 In ProgressbkingT427306 Migrate Relforge clusters from OpenSearch 1.x->2x
 In ProgressatsukoT427517 Issue 6 month certificates for OpenSearch-on-K8S

Event Timeline

bking created this task.Mar 30 2026, 6:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2026, 6:00 PM
bking added a project: Discovery-Search (2026.03.03 - 2026.04.03).Mar 30 2026, 6:00 PM
bking updated the task description. (Show Details)Mar 30 2026, 6:14 PM
bking created subtask T421763: Migrate beta cluster to OpenSearch 2.x.Mar 30 2026, 6:27 PM
bking updated the task description. (Show Details)Mar 30 2026, 8:45 PM
bking updated the task description. (Show Details)Mar 30 2026, 9:15 PM
bking changed the status of subtask T421763: Migrate beta cluster to OpenSearch 2.x from Open to In Progress.Apr 9 2026, 3:43 PM
bking created subtask T422860: Migrate Cloudelastic to OpenSearch 2.x.Apr 9 2026, 4:35 PM
bking closed subtask T421763: Migrate beta cluster to OpenSearch 2.x as Declined.Apr 9 2026, 4:41 PM
pfischer edited projects, added Discovery-Search (2026.04.06 - 2026.05.01); removed Discovery-Search (2026.03.03 - 2026.04.03).Apr 13 2026, 1:58 PM
bking updated the task description. (Show Details)Apr 20 2026, 10:11 PM

An early and incomplete list of indices that need to be recreated is here .
Note that these are mostly archive, titlesuggest, and non-cirrus indices like toolhub_tools and ttmserver. We'll need to inform stakeholders and help them migrate as part of this process.

dcausse created subtask T423993: Upgrade old indices in the CirrusSearch opensearch clusters.Apr 21 2026, 8:32 AM
bking updated the task description. (Show Details)Apr 21 2026, 7:40 PM
bking reopened subtask T421763: Migrate beta cluster to OpenSearch 2.x as In Progress.
bking created subtask T424248: ☂️Migrate non-cirrus indices from production OpenSearch to OpenSearch on k8s ☂️.Apr 23 2026, 5:05 PM
Gehel edited projects, added Data-Platform-SRE (2026-04-24 - 2026-05-15); removed Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:17 AM
bking created subtask T424852: Investigate performance issues in cloudelastic.Apr 29 2026, 1:56 PM
bking changed the status of subtask T424852: Investigate performance issues in cloudelastic from Open to In Progress.Apr 29 2026, 2:38 PM
atsuko changed the status of subtask T424248: ☂️Migrate non-cirrus indices from production OpenSearch to OpenSearch on k8s ☂️ from Open to In Progress.Apr 29 2026, 4:04 PM
bking closed subtask T422860: Migrate Cloudelastic to OpenSearch 2.x as Resolved.Apr 29 2026, 7:53 PM
bking created subtask T426101: Create detailed migration plan for OpenSearch 1.x-2x.Tue, May 12, 4:57 PM
pfischer edited projects, added Discovery-Search (2026.05.04 - 2026.05.29); removed Discovery-Search (2026.04.06 - 2026.05.01).Mon, May 18, 1:43 PM
bking updated the task description. (Show Details)Tue, May 19, 2:38 PM
bking updated the task description. (Show Details)Wed, May 20, 1:42 PM
bking created subtask T426862: Ensure cookbooks work with OpenSearch 2.x.Wed, May 20, 1:51 PM
bking changed the status of subtask T426862: Ensure cookbooks work with OpenSearch 2.x from Open to In Progress.Wed, May 20, 1:54 PM
bking changed the status of subtask T426101: Create detailed migration plan for OpenSearch 1.x-2x from Open to In Progress.Wed, May 20, 2:59 PM
dcausse moved this task from Incoming to Blocked / Waiting on the Discovery-Search (2026.05.04 - 2026.05.29) board.Fri, May 22, 1:44 PM
bking updated the task description. (Show Details)Tue, May 26, 1:12 PM
bking updated the task description. (Show Details)
bking changed the status of subtask T423993: Upgrade old indices in the CirrusSearch opensearch clusters from Open to In Progress.Tue, May 26, 2:02 PM
bking updated the task description. (Show Details)Tue, May 26, 5:13 PM
bking created subtask T427306: Migrate Relforge clusters from OpenSearch 1.x->2x.Tue, May 26, 5:23 PM
bking changed the status of subtask T427306: Migrate Relforge clusters from OpenSearch 1.x->2x from Open to In Progress.Tue, May 26, 6:30 PM
bking updated the task description. (Show Details)Wed, May 27, 1:25 PM
gerritbot added a comment.Wed, May 27, 8:25 PM

Change #1294402 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] OpenSearch: Add required config for bootstrapping a cluster

https://gerrit.wikimedia.org/r/1294402

gerritbot added a project: Patch-For-Review.Wed, May 27, 8:26 PM
bking updated the task description. (Show Details)Wed, May 27, 10:04 PM
atsuko added a subtask: T427517: Issue 6 month certificates for OpenSearch-on-K8S.Thu, May 28, 2:04 PM
bking changed the status of subtask T427517: Issue 6 month certificates for OpenSearch-on-K8S from Open to In Progress.Thu, May 28, 9:57 PM
gerritbot added a comment.Fri, May 29, 1:10 PM

Change #1294402 merged by Bking:

[operations/puppet@production] OpenSearch: Add required config for bootstrapping

https://gerrit.wikimedia.org/r/1294402

Maintenance_bot removed a project: Patch-For-Review.Fri, May 29, 1:31 PM
Gehel edited projects, added Data-Platform-SRE (2026-06-05 - 2026-06-26); removed Data-Platform-SRE (2026-04-24 - 2026-05-15).Fri, Jun 5, 1:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits