Maniphest T389812

cirrussearch-opensearch-image should provide an opensearch image that offers the same functionnalities as the production opensearch cluster
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	dcausse
	Mar 24 2025, 2:50 PM

Project Tags

Referenced Files

None

Subscribers

Description

cirrussearch-opensearch-image is built on top the official opensearch image which is including additional plugins.
These plugins are not installed on production opensearch hosts.
This can possibly be quite error-prone since a developer & cindy might accidentally rely on features not available in production to develop & test CirrusSearch.

AC:

decide on an approach
- find another base image
- build our own base image using our own opensearch deb
- hack the official base image and remove the additional plugins as part of the blubber build
cirrussearch-opensearch-image & the opensearch production cluster provide the same set of plugins

Details

Other Assignee: EBernhardson

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Add opensearch-knn	DCausse	operations/software/opensearch/plugins	master	+24 -1

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Update opensearch plugins .deb to 1.3.20-3	repos/search-platform/cirrussearch-opensearch-image!13	ebernhardson	work/ebernhardson/knn	main
	Remove additional plugins available in the opensearch base image	repos/search-platform/cirrussearch-opensearch-image!12	dcausse	T389812-remove-pre-installed-plugins	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T370147 [Epic] Migration of the Elasticsearch 7.10 search cluster to OpenSearch
		Resolved		dcausse	T389812 cirrussearch-opensearch-image should provide an opensearch image that offers the same functionnalities as the production opensearch cluster
		Resolved		bking	T390100 Build and deploy updated opensearch plugins deb

Event Timeline

dcausse created this task.Mar 24 2025, 2:50 PM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptMar 24 2025, 2:50 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Gehel triaged this task as High priority.Mar 24 2025, 3:59 PM

Gehel edited projects, added: Discovery-Search (2025.03.01 - 2025.03.21); removed: Discovery-Search.

bking subscribed.Mar 24 2025, 4:28 PM

Gehel moved this task from Incoming to Ready for Dev on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 24 2025, 4:29 PM

Gehel set the point value for this task to 3.

Gehel edited projects, added: Discovery-Search (2025.03.22 - 2025.04.11); removed: Discovery-Search (2025.03.01 - 2025.03.21).Mar 24 2025, 4:50 PM

Gehel moved this task from Incoming to Ready for Dev on the Discovery-Search (2025.03.22 - 2025.04.11) board.

dcausse claimed this task.Mar 25 2025, 9:46 AM

dcausse moved this task from Ready for Dev to In Progress on the Discovery-Search (2025.03.22 - 2025.04.11) board.

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/merge_requests/12

Remove additional plugins available in the opensearch base image

dcausse moved this task from In Progress to Needs Review on the Discovery-Search (2025.03.22 - 2025.04.11) board.Mar 25 2025, 2:19 PM

Should we also add the knn plugin to the .deb? I suppose thats not exactly what this ticket is about, but it's what prompted this ticket

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/merge_requests/12

Remove additional plugins available in the opensearch base image

@EBernhardson definitely I'll get patch opened

Maintenance_bot removed a project: Patch-For-Review.Mar 25 2025, 5:31 PM

Change #1131068 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/software/opensearch/plugins@master] Add opensearch-knn

https://gerrit.wikimedia.org/r/1131068

gerritbot added a project: Patch-For-Review.Mar 25 2025, 6:07 PM

Change #1131286 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/mediawiki-config@master] wdqs: enable hive/hdfs ingestion for rdf update streams

https://gerrit.wikimedia.org/r/1131286

ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/merge_requests/13

Remove additional plugins available in the opensearch base image

bking changed the status of subtask T390100: Build and deploy updated opensearch plugins deb from Open to In Progress.Mar 27 2025, 2:34 PM

Change #1131068 merged by Bking:

[operations/software/opensearch/plugins@master] Add opensearch-knn

https://gerrit.wikimedia.org/r/1131068

dcausse mentioned this in rOSOPd61e3e1b00a4: Add opensearch-knn.Mar 27 2025, 2:39 PM

Moving to To be deployed waiting for T390100 to be done.

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/merge_requests/13

Update opensearch plugins .deb to 1.3.20-3

It took a couple tries, but this is now released as 1.3.20-5

In T389812#10694730, @EBernhardson wrote:

It took a couple tries, but this is now released as 1.3.20-5

Thanks!

Gehel added a parent task: T370147: [Epic] Migration of the Elasticsearch 7.10 search cluster to OpenSearch.Apr 4 2025, 9:35 AM

Gehel moved this task from Done to Reported on the Discovery-Search (2025.03.22 - 2025.04.11) board.

bking closed subtask T390100: Build and deploy updated opensearch plugins deb as Resolved.Apr 30 2025, 9:35 PM