Page MenuHomePhabricator
Create Task
Maniphest T370147

[Epic] Migration of the Elasticsearch 7.10 search cluster to OpenSearch
Open, MediumPublic
Actions

Description

As discussed in T272111, Elasticsearch has switched license and is no longer an option in our context. Migration to OpenSearch needs to happen to benefit from new features in the search space, to keep up with technical debt and to run supported software for security and operability reasons.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT370147 [Epic] Migration of the Elasticsearch 7.10 search cluster to OpenSearch
 ResolvedGehelT370148 Create high level plan for the migration of the Elasticsearch 7.10 search cluster to replacement backend search engine
 Resolveddr0ptp4ktT370661 Decision brief about OpenSearch vs Elasticsearch
 ResolvedEBernhardsonT372769 Adjust cirrus-integration-test-runner to boot opensearch
 ResolvedbkingT380395 Build/release opensearch-plugins Debian package
 OpenNoneT379288 Plan for access control with opensearch
 ResolvedGehelT379938 Evaluate mixed-cluster behavior of elasticsearch + opensearch
 ResolvedbkingT379312 Release packages for opensearch 1.3.20
 ResolveddcausseT385005 Build all search platform plugins for opensearch 1.3.20
 ResolvedbkingT380752 Migrate Relforge to Opensearch
 ResolvedbkingT386281 Decide if we still need opensearch-madvise
 ResolvedbkingT386357 Replace current Relforge servers with repurposed Elastic hosts
 ResolvedVRiley-WMFT386358 Relabel Elastic hosts to Relforge hosts
 DeclinedbkingT386730 Create thirdparty/opensearch13 repo for bullseye and remove thirdparty/opensearch1 repo
 OpenNoneT386525 Replace curator with opensearch index state management
 In ProgressbkingT386860 Enable CPU performance governor on Relforge, Cloudelastic, and Elasticsearch hosts
 ResolvedbkingT387028 Decide on a new name for Elastic hosts/Rename hosts during reimage
 ResolvedTJonesT386870 Regression Test OpenSearch Language Analysis
 ResolvedEBernhardsonT386868 Port Sudachi to OpenSearch 1.x
 ResolvedRKemperT388595 Release new deb from opensearch plugins repository
 ResolvedEBernhardsonT389119 Upgrade wmf_opensearch_search_plugins .deb and restart opensearch
 ResolvedbkingT387904 Migrate Cloudelastic to Opensearch
 ResolvedbkingT388150 cloudelastic1008 stuck at boot screen after multiple reboots, SEL reports Comm Error: Backplane 0
 ResolvedbkingT389128 Update WMCS docs to reflect Cloudelastic's migration from Elastic -> OpenSearch
 ResolvedbkingT388610 Migrate production Elastic clusters to Opensearch
 DeclinedNoneT383811 Ensure Search Platform-owned Elasticsearch cookbooks can handle Opensearch
 ResolvedbkingT391151 Ensure our automation can ban not-yet-existing hosts
 ResolvedPRODUCTION ERRORdcausseT390244 InvalidArgumentException: Default TTM service eqiad cannot be write only
 ResolvedbkingT391350 Review/rollback all temporary changes from the OpenSearch migration
 ResolvedbkingT391351 Test OpenSearch cluster behavior when we add non-existent hosts to static config
 ResolvedJhancock.wmT391639 Comm Error: Backplane 0 on cirrussearch2091 (Row/Rack A7)
 ResolvedPapaulT392644 cirrussearch2078 (R440 Config D, Row/Rack B2) unable to PXE boot
 ResolvedbkingT393100 Fix cross-cluster seed settings in Search Platform production clusters
 ResolvedbkingT395331 Find/ban "ghost" Elastic hosts
 ResolvedGehelT388607 Remove unused Elastic-specific code from Puppet/Spicerack/Cookbooks after Opensearch migration is complete
 ResolveddcausseT386945 The export_queries_to_relforge dag is not compatible with opensearch
 ResolvedEBernhardsonT388686 Correctly report OpenSearch/Elasticsearch to Special:Version
 ResolvedEBernhardsonT388611 Add version checking to wmf-search opensearch plugins repo
 ResolvedGehelT389092 Update documentation to reflect the migration from Elasticsearch to OpenSearch
 ResolvedbkingT389971 Migrate deployment-prep elasticsearch cluster to opensearch
 ResolveddcausseT389812 cirrussearch-opensearch-image should provide an opensearch image that offers the same functionnalities as the production opensearch cluster
 ResolvedbkingT390100 Build and deploy updated opensearch plugins deb

Event Timeline

Gehel created this task.Jul 16 2024, 12:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2024, 12:50 PM
TehKittyCat subscribed.Jul 18 2024, 2:42 PM
Gehel triaged this task as Medium priority.Jul 22 2024, 1:44 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel moved this task from Incoming to Epics on the Discovery-Search (Current work) board.
Gehel closed subtask T370661: Decision brief about OpenSearch vs Elasticsearch as Resolved.Aug 23 2024, 12:53 PM
bking reopened subtask T370661: Decision brief about OpenSearch vs Elasticsearch as Open.Aug 30 2024, 2:12 PM
Jaideraf subscribed.Sep 5 2024, 11:19 AM
Gehel added a comment.Sep 6 2024, 12:22 PM

Given the recent changes to Elasticsearch license to AGPL, we are likely going to stay on Elasticsearch and upgrade to latest version. Final decision to come...

dr0ptp4kt renamed this task from [Epic] Migration of the Elasticsearch search cluster to OpenSearch to [Epic] Migration of the Elasticsearch 7.10 search cluster to replacement backend search engine.Sep 9 2024, 8:05 PM
dr0ptp4kt closed subtask T370661: Decision brief about OpenSearch vs Elasticsearch as Resolved.Oct 11 2024, 10:47 AM
• XiaoXiao-WMF added a project: Research.Oct 29 2024, 3:52 PM
• XiaoXiao-WMF moved this task from Backlog to Watching on the Research board.
EBernhardson added a subtask: T379312: Release packages for opensearch 1.3.20.Nov 14 2024, 6:12 PM
Gehel closed subtask T379938: Evaluate mixed-cluster behavior of elasticsearch + opensearch as Resolved.Nov 15 2024, 9:48 AM
bking subscribed.Nov 15 2024, 3:47 PM
gmodena subscribed.Nov 21 2024, 9:30 AM
Gehel closed subtask T372769: Adjust cirrus-integration-test-runner to boot opensearch as Resolved.Dec 6 2024, 1:24 PM
bking added a comment.Jan 14 2025, 5:49 PM

Following up on @Gehel 's last comment. Search Platform team (along with stakeholders) has officially decided to migrate to Opensearch. See T370661 for more details about how any why we made that decision.

bking added a subtask: T383811: Ensure Search Platform-owned Elasticsearch cookbooks can handle Opensearch.Jan 16 2025, 2:19 PM
Gehel closed subtask T370148: Create high level plan for the migration of the Elasticsearch 7.10 search cluster to replacement backend search engine as Resolved.Jan 24 2025, 1:17 PM
bking changed the status of subtask T379312: Release packages for opensearch 1.3.20 from Open to In Progress.Jan 27 2025, 6:01 PM
Gehel edited projects, added Discovery-Search (2025.02.10 - 2025.02.28); removed Discovery-Search (Current work).Feb 11 2025, 2:20 PM
Gehel moved this task from Incoming to Epics on the Discovery-Search (2025.02.10 - 2025.02.28) board.
brouberol closed subtask T379312: Release packages for opensearch 1.3.20 as Resolved.Feb 12 2025, 8:11 AM
bking mentioned this in T386860: Enable CPU performance governor on Relforge, Cloudelastic, and Elasticsearch hosts.Feb 19 2025, 6:54 PM
bking added a subtask: T386860: Enable CPU performance governor on Relforge, Cloudelastic, and Elasticsearch hosts.Feb 19 2025, 8:22 PM
bking mentioned this in T365814: Test whether or not CPU performance governor helps Elasticsearch performance.Feb 20 2025, 1:16 AM
bking changed the status of subtask T383811: Ensure Search Platform-owned Elasticsearch cookbooks can handle Opensearch from Open to In Progress.Feb 20 2025, 7:12 PM
bking changed the status of subtask T386860: Enable CPU performance governor on Relforge, Cloudelastic, and Elasticsearch hosts from Open to In Progress.Feb 20 2025, 9:17 PM
bking changed the status of subtask T380752: Migrate Relforge to Opensearch from Open to In Progress.Feb 20 2025, 9:38 PM
bking mentioned this in T386983: Explore migrating wdqs' nginx config into envoy.Feb 20 2025, 10:53 PM
bking mentioned this in T387028: Decide on a new name for Elastic hosts/Rename hosts during reimage.Feb 21 2025, 2:28 PM
bking added a subtask: T387028: Decide on a new name for Elastic hosts/Rename hosts during reimage.
Krinkle mentioned this in T272111: Elasticsearch, a CirrusSearch dependency, is switching to SSPL/Custom licence.Feb 22 2025, 4:48 PM
Gehel edited projects, added Discovery-Search (2025.03.01 - 2025.03.21); removed Discovery-Search (2025.02.10 - 2025.02.28).Feb 28 2025, 1:38 PM
Gehel moved this task from Incoming to Epics on the Discovery-Search (2025.03.01 - 2025.03.21) board.
Gehel added a subtask: T386870: Regression Test OpenSearch Language Analysis.Mar 3 2025, 2:44 PM
Gehel added a subtask: T386868: Port Sudachi to OpenSearch 1.x.
bking changed the status of subtask T380752: Migrate Relforge to Opensearch from In Progress to Open.Mar 4 2025, 10:01 PM
Krinkle added a subtask: T387904: Migrate Cloudelastic to Opensearch.Mar 4 2025, 10:20 PM
Gehel edited projects, added Discovery-Search; removed Discovery-Search (2025.03.01 - 2025.03.21).Mar 5 2025, 10:22 AM
Gehel moved this task from 2025.03.01 - 2025.03.21 to Active Epics on the Discovery-Search board.
bking added a subtask: T388607: Remove unused Elastic-specific code from Puppet/Spicerack/Cookbooks after Opensearch migration is complete.Mar 13 2025, 2:24 PM
leila removed a project: Research.Mar 13 2025, 4:59 PM
bking changed the status of subtask T388610: Migrate production Elastic clusters to Opensearch from Open to In Progress.Mar 17 2025, 2:39 PM
Gehel added subtasks: T386945: The export_queries_to_relforge dag is not compatible with opensearch, T388686: Correctly report OpenSearch/Elasticsearch to Special:Version.Mar 17 2025, 3:13 PM
Gehel added a subtask: T388611: Add version checking to wmf-search opensearch plugins repo.
dcausse closed subtask T386945: The export_queries_to_relforge dag is not compatible with opensearch as Resolved.Mar 17 2025, 5:21 PM
bking changed the status of subtask T388610: Migrate production Elastic clusters to Opensearch from In Progress to Stalled.Mar 18 2025, 7:42 PM
bking closed subtask T380752: Migrate Relforge to Opensearch as Resolved.Mar 18 2025, 9:15 PM
bking changed the status of subtask T388610: Migrate production Elastic clusters to Opensearch from Stalled to In Progress.Mar 19 2025, 1:39 PM
bking mentioned this in T384966: Q3:rack/setup/install elastic1111-elastic1122, relforge1008-1010.Mar 19 2025, 5:23 PM
bking reopened subtask T380752: Migrate Relforge to Opensearch as In Progress.Mar 20 2025, 3:52 PM
bking closed subtask T380752: Migrate Relforge to Opensearch as Resolved.Mar 21 2025, 1:45 PM
Gehel closed subtask T388611: Add version checking to wmf-search opensearch plugins repo as Resolved.Mar 21 2025, 1:57 PM
Gehel closed subtask T386870: Regression Test OpenSearch Language Analysis as Resolved.
bking closed subtask T387028: Decide on a new name for Elastic hosts/Rename hosts during reimage as Resolved.Mar 25 2025, 2:46 PM
bking added a subtask: T389971: Migrate deployment-prep elasticsearch cluster to opensearch.Mar 25 2025, 7:56 PM
bking changed the status of subtask T389971: Migrate deployment-prep elasticsearch cluster to opensearch from Open to In Progress.Mar 26 2025, 1:44 PM
Gehel closed subtask T386868: Port Sudachi to OpenSearch 1.x as Resolved.Mar 28 2025, 2:16 PM
Gehel closed subtask T389971: Migrate deployment-prep elasticsearch cluster to opensearch as Resolved.Apr 4 2025, 7:44 AM
Gehel added a subtask: T389812: cirrussearch-opensearch-image should provide an opensearch image that offers the same functionnalities as the production opensearch cluster.Apr 4 2025, 9:35 AM
Gehel closed subtask T388686: Correctly report OpenSearch/Elasticsearch to Special:Version as Resolved.Apr 4 2025, 9:39 AM
bking added a subtask: T391350: Review/rollback all temporary changes from the OpenSearch migration.Apr 8 2025, 1:51 PM
bking closed subtask T387904: Migrate Cloudelastic to Opensearch as Resolved.Apr 14 2025, 1:52 PM
Diskdance renamed this task from [Epic] Migration of the Elasticsearch 7.10 search cluster to replacement backend search engine to [Epic] Migration of the Elasticsearch 7.10 search cluster to OpenSearch.May 5 2025, 2:53 PM
Jdforrester-WMF reopened subtask T388610: Migrate production Elastic clusters to Opensearch as Open.May 23 2025, 6:54 PM
Novem_Linguae subscribed.May 28 2025, 5:15 PM
KeybladeSpyMaster subscribed.Jun 9 2025, 3:16 PM
• roti_WMDE subscribed.Jun 16 2025, 7:14 AM
EBernhardson mentioned this in T394295: Unexpected changes in search latency.Jun 24 2025, 5:40 PM
bking closed subtask T388610: Migrate production Elastic clusters to Opensearch as Resolved.Jun 27 2025, 1:15 PM
bking closed subtask T391350: Review/rollback all temporary changes from the OpenSearch migration as Resolved.Jul 7 2025, 7:43 PM
bking changed the status of subtask T388607: Remove unused Elastic-specific code from Puppet/Spicerack/Cookbooks after Opensearch migration is complete from Open to In Progress.Jul 29 2025, 2:25 PM
BTullis mentioned this in T403301: Discuss OpenSearch 3 roadmap/future improvements.Sep 19 2025, 11:02 AM
Gehel closed subtask T389092: Update documentation to reflect the migration from Elasticsearch to OpenSearch as Resolved.Oct 17 2025, 12:33 PM
Gehel closed subtask T383811: Ensure Search Platform-owned Elasticsearch cookbooks can handle Opensearch as Declined.Wed, Feb 4, 8:32 AM
Gehel closed subtask T388607: Remove unused Elastic-specific code from Puppet/Spicerack/Cookbooks after Opensearch migration is complete as Resolved.Wed, Feb 4, 10:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits