Explore options for OpenSearch 2.x/3.x plugin packaging and distribution
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bking
	Tue, Apr 14, 6:22 PM

Description

We (Data Platform SRE and Search platform team) have had several issues with OpenSearch packaging since moving to OpenSearch 2 and 3. We need to decide on a packaging strategy.

We started out using an OpenSearch 2.7 from a repo created WMF's Observability team. We had to abandon this due to the age of the package .

Next, we switched to the OpenSearch project's latest upstream 2.x and 3.x packages. While this has kept us current in terms of features and security, there are some drawbacks here as well:

Forced installation of security demo, this comment sums it up pretty well. We are optimistic that we can engage with the upstream and move this forward, but the Github issue has been around for awhile.

Large number of plugins installed by default Per IRC conversation with @dcausse , the upstream package enables 26 plugins by default. For security and performance reasons, we want to be more intentional about which plugins we activate.

Currently, the Search Platform removes all plugins from its Docker image , but this is causing issues where we end up removing the plugins here, then re-adding them into our wmf-opensearch-search-plugins package. This is causing the package to fail installation, as the OpenSearch package already owns these packages:

dpkg: error processing archive /var/cache/apt/archives/wmf-opensearch-search-plugins_2.19.5+3~bullseye_all.deb (--unpack):
 trying to overwrite '/usr/share/opensearch/plugins/opensearch-knn/LICENSE.txt', which is also in package opensearch (2.19.5)

One tantalizing option is to use the opensearch-min tarball as a source of a Debian package.

However, this gets us farther away from upstream and I would argue that we should try to engage with upstream and try to make improvements there if possible. We could also just continue using the upstream package and disable the plugins we don't want.

Creating this ticket to:

Discuss requirements with stakeholders
Determine level of effort for disabling plugins with Puppet or some other automation
Install the opensearch-min tarball outside of containers and determine its viability as a source for a deb package. If it seems reasonable, we could submit a PR to build the deb upstream (no guarantee they would accept it though). declined, not needed

Once we have done the above, we should be able to decide on how to move forward:

Keep using upstream packages, but disable the unwanted plugins with Puppet or some other automation
~~Use the opensearch-min tarball to build our own Debian packages and attempt to get these changes merged upstream.~~ declined

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
cloudelastic1012: Set LVS config for opensearch_2	operations/puppet	production	+13 -0
cloudelastic1012: full common_settings override for OS2	operations/puppet	production	+11 -4
cloudelastic1012: full common_settings override for OS2	operations/puppet	production	+17 -2
cloudelastic1012: full common_settings override for OS2	operations/puppet	production	+17 -2
cloudelastic1012: remove the deliberately-introduced typo	operations/puppet	production	+1 -1
opensearch: move var up so we can use it earlier	operations/puppet	production	+4 -2
OpenSearch: Control which plugins we use via systemd PrivateMounts	operations/puppet	production	+29 -0
opensearch: strip bundled plugins before WMF pkg	operations/puppet	production	+25 -0
opensearch: allowlist upstream plugins + overwrite	operations/puppet	production	+54 -13
cloudelastic: temporarily add "working typos" for plugins	operations/puppet	production	+17 -0
opensearch: remove dead plugins_dir parameter	operations/puppet	production	+0 -7

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Remove already-installed packages	repos/search-platform/opensearch-plugins-deb!20	bking	remove_packages	master

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Open	None	T421757 ☂️ Migrate production OpenSearch clusters from 1.x-2.x ☂️
Resolved	bking	T422860 Migrate Cloudelastic to OpenSearch 2.x
Resolved	bking	T423327 Explore options for OpenSearch 2.x/3.x plugin packaging and distribution
Resolved	bking	T423523 Handle typos/possibly update opensearch-analysis-stconvert plugin

Event Timeline

bking created this task.Tue, Apr 14, 6:22 PM

bking triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Apr 14, 6:22 PM

bking renamed this task from Explore options for opensearch 2.x/trixie packaging to Explore options for OpenSearch 2.x/3.x packaging and distribution.Tue, Apr 14, 8:38 PM

bking edited projects, added Data-Platform-SRE (2026-03-27 - 2026-04-17); removed Data-Platform-SRE.

bking updated the task description. (Show Details)

bking added a subscriber: dcausse.

bking updated the task description. (Show Details)Tue, Apr 14, 8:44 PM

bking updated the task description. (Show Details)

bking added a subscriber: EBernhardson.

The following plugins are currently active in cirrussearch production (the OpenSearch clusters that support production wikipedia)

analysis-hebrew
analysis-icu
analysis-nori
analysis-smartcn
analysis-stempel
analysis-sudachi
analysis-ukrainian
cirrus-highlighter
ltr
opensearch-analysis-stconvert
opensearch-extra
opensearch-extra-analysis-esperanto
opensearch-extra-analysis-homoglyph
opensearch-extra-analysis-khmer
opensearch-extra-analysis-serbian
opensearch-extra-analysis-slovak
opensearch-extra-analysis-textify
opensearch-extra-analysis-turkish
opensearch-extra-analysis-ukrainian
opensearch-knn
repository-s3

Source: curl http://0:9200/_cat/plugins | sort -k2 | awk '{print $2}' | uniq from a cirrusssearch host

I've also confirmed all plugins are provided by the WMF's own wmf-opensearch-search-plugins package via for n in $(cat /tmp/plugins); do echo ${n}; dpkg -L wmf-opensearch-search-plugins | grep -x /usr/share/opensearch/plugins/${n}; done

For contrast, here is what we deploy on opensearch-ipoid (OpenSearch 2.19.4 from the upstream Debian package with the prometheus-exporter plugin added):

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
prometheus-exporter
query-insights

And opensearch-semantic-search (OpenSearch 3.5.0 from the upstream Debian package, with the prometheus-exporter plugin added)

analysis-icu
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-inner-hits
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-search-relevance
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
opensearch-ubi
prometheus-exporter
query-insights

The next step is figuring out level of effort for disabling the unwanted plugins. Can we just remove the files, or will OpenSearch scream when we restart? If the files are owned by a package, how can we get them to say away? Would it be safer to clobber the current plugins directory instead of deleting files?

bking updated the task description. (Show Details)Tue, Apr 14, 9:19 PM

Change #1271473 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] opensearch: strip bundled plugins before WMF pkg

https://gerrit.wikimedia.org/r/1271473

gerritbot added a project: Patch-For-Review.Wed, Apr 15, 7:02 AM

We should def work with upstream, but let's be pessimistic about timing, so we should patch on our end too.

I did some digging, I think there's two solutions here. The more robust one requires a package rebuild, but is otherwise pretty simple.

Quick fix:

I've got a patch that mirrors what the Docker image does: after opensearch installs/upgrades, strip all bundled plugins, then reinstall wmf-opensearch-search-plugins to restore the correct set. Two execs:

(1) strip-upstream-opensearch-plugins — subscribed to Package['opensearch'], uses find -mindepth 1 -delete to clear the plugins dir. Only fires when the opensearch package actually changes

(2) reinstall-wmf-opensearch-plugins — runs on every puppet apply but only acts when the plugins dir is empty (onlyif guard). Handles the upgrade case where the WMF package is already "present" so puppet would otherwise no-op, and self-heals after interrupted runs

Note it doesn't handle the case where opensearch 2.19+ is already installed before this puppet code lands, but since we're reimaging for the migration that should be fine

Proper fix:

OpenSearch supports configuring path.plugins in opensearch.yml. If we set it to something like /usr/share/opensearch/wmf-plugins and rebuild wmf-opensearch-search-plugins to deploy there, we sidestep the dpkg conflict entirely. opensearch ignores the upstream-bundled plugins in the default path and only loads from our custom path. No strip/reinstall dance, no dpkg conflicts, survives upgrades cleanly.

This requires rebuilding the WMF plugins deb to deploy to the new path, so it's more work upfront but eliminates the whole class of problem. Also solves the "26 unwanted plugins" concern since opensearch just won't load them.

In T423327#11822826, @RKemper wrote:

Proper fix:

OpenSearch supports configuring path.plugins in opensearch.yml. If we set it to something like /usr/share/opensearch/wmf-plugins and rebuild wmf-opensearch-search-plugins to deploy there, we sidestep the dpkg conflict entirely. opensearch ignores the upstream-bundled plugins in the default path and only loads from our custom path. No strip/reinstall dance, no dpkg conflicts, survives upgrades cleanly.

This seems like a nice trick to prevent opensearch from loading its default plugins.

This requires rebuilding the WMF plugins deb to deploy to the new path, so it's more work upfront but eliminates the whole class of problem. Also solves the "26 unwanted plugins" concern since opensearch just won't load them.

Unless I'm missing something this sounds like a viable solution.

For search the requirements are mainly:

own the list of plugin and don't have a plugin installed just because it's part of the vanilla install
use the jvm from the wmf maintained package (sec updates/proper maintenance of the keystore for certs) and delete/ignore the one included in the package

Thanks for sharing your feedback, here's some more of my own!

OpenSearch supports configuring path.plugins in opensearch.yml. If we set it to something like /usr/share/opensearch/wmf-plugins and rebuild wmf-opensearch-search-plugins to deploy there, we sidestep the dpkg conflict entirely. opensearch ignores the upstream-bundled plugins in the default path and only loads from our custom path. No strip/reinstall dance, no dpkg conflicts, survives upgrades cleanly.

I can't find any reference to path.plugins in official OpenSearch docs. I think that's how Elasticsearch used to handle plugins before the opensearch-plugin tool existed. Unfortunately, I don't think it's an option anymore.

But I do think we can accomplish @dcausse /Search team's requirements. Per David & myself's pairing today, we talked about a few changes we could make:

Search platform using the WMF OpenSearch docker images or building on top of them for their tests. Since those images are built on Debian, it should be more production-like than the current image which is based on Amazon Linux.

Compile a list of needed plugins for Search platform and other DPE-SRE supported users (such as ipoid) and use them to create an allow list. For Search Platform, I believe the only OpenSearch-bundled plugins needed are listed here . As far as I know, all other plugins required by Search Platform are shipped in the wmf-opensearch-search-plugins package.

Once Search Platform has transitioned to the WMF OpenSearch image, remove the above plugins from the wmf-opensearch-search-plugins package.

Figure out a way to manage the plugins directory with Puppet or some other orchestration.

bking renamed this task from Explore options for OpenSearch 2.x/3.x packaging and distribution to Explore options for OpenSearch 2.x/3.x plugin packaging and distribution.Wed, Apr 15, 3:13 PM

bking merged a task: T423291: Build new wmf-opensearch-search-plugins package for opensearch 2.x/trixie and ensure we don't install/enable any unwanted plugins in prod.

bking mentioned this in T423291: Build new wmf-opensearch-search-plugins package for opensearch 2.x/trixie and ensure we don't install/enable any unwanted plugins in prod.

bking updated the task description. (Show Details)Wed, Apr 15, 3:17 PM

Change #1271473 merged by Bking:

[operations/puppet@production] opensearch: strip bundled plugins before WMF pkg

https://gerrit.wikimedia.org/r/1271473

bking merged https://gitlab.wikimedia.org/repos/search-platform/opensearch-plugins-deb/-/merge_requests/20

Remove already-installed packages

I can't find any reference to path.plugins in official OpenSearch docs. I think that's how Elasticsearch used to handle plugins before the opensearch-plugin tool existed. Unfortunately, I don't think it's an option anymore.

Confirmed, I enabled this setting and attempted to start the OpenSearch 2 service on cloudelastic1012; it erred:

Apr 15 17:33:41 cloudelastic1012 opensearch[937279]: java.lang.IllegalArgumentException: unknown setting [path.plugins] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Our current Puppet code still has a reference to this setting; we should remove it to prevent confusion.

Change #1271843 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] opensearch: remove dead plugins_dir parameter

https://gerrit.wikimedia.org/r/1271843

Great catch on that Brian. I uploaded a patch to remove the stale references in puppet.

wrt the issue you posted on slack:

Apr 15 17:19:47 cloudelastic1012 opensearch[935010]: java.lang.IllegalStateException: missing mandatory plugins [ltr, opensearch-analysis-stconvert], found plugins [opensearch-extra-analysis-serbian, analysis-ukrainian, opensearch-extra-analysis-homoglyph, opensearch-extra-analysis-khmer, opensearch-extra-analysis-ukrainian, opensearch-analisys-stconvert, opensearch-extra-analysis-esperanto, opensearch-extra-analysis-slovak, analysis-hebrew, opensearch-extra-analysis-textify, repository-s3, analysis-icu, analysis-sudachi, opensearch-extra-analysis-turkish, analysis-smartcn, opensearch-extra, analysis-stempel, cirrus-highlighter, analysis-nori]

Two issues:

(1) opensearch-analysis-stconvert ("analisys" typo)

The typo was in the download URL in the opensearch-plugins-deb repo (opensearch-analisys-stconvert instead of opensearch-analysis-stconvert). dcausse fixed the download URL in commit 4d6bec8 (Feb 2025), but the issue was reintroduced in af9d932 (Mar 2026), so the current Trixie plugins package (2.19.5+4~bullseye) still has the typo'd directory name on disk.

TLDR: We need to re-apply the fix and rebuild the plugins package.

(2) ltr missing

The LTR plugin went through a transition for OS 2.19.5 — it was temporarily removed during the "Update plugins for opensearch 2.19.5" commit (af9d932, Mar 2026), then re-added using the official OpenSearch build (opensearch-ltr-2.19.5.0) instead of the old WMF custom build (ltr-1.5.4-wmf1). But the plugin registers as opensearch-ltr in the official build, while the mandatory plugins list in hieradata expects ltr. Name mismatch.

So presumably we just need to update the hieradata mandatory plugins list to use opensearch-ltr

bking created subtask T423523: Handle typos/possibly update opensearch-analysis-stconvert plugin.Wed, Apr 15, 8:12 PM

Change #1271843 merged by Bking:

[operations/puppet@production] opensearch: remove dead plugins_dir parameter

https://gerrit.wikimedia.org/r/1271843

Change #1271929 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: temporarily add "working typos" for plugins

https://gerrit.wikimedia.org/r/1271929

Change #1271929 merged by Bking:

[operations/puppet@production] cloudelastic: temporarily add "working typos" for plugins

https://gerrit.wikimedia.org/r/1271929

Change #1271947 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] opensearch: allowlist upstream-only plugins

https://gerrit.wikimedia.org/r/1271947

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/opensearch-analysis-stconvert/-/merge_requests/9

Fix stconvert typo & restore gradle publish URL

Copying my comment here too, now that I've seen this task:

Something to consider:

The packaging and deployment of opensearch dashboards plugins via debs was painful enough for us to write a puppet provider[0] for them. The puppet provider is fairly simple and it looks to me like OpenSearch's plugin management is similar in form[1]. We have a repo (really, just a dir) on apt.wm.o specifically for opensearch and dashboards plugins if copying the dashboards provider into an opensearch-specific one would be useful for your use case.

[0] https://gerrit.wikimedia.org/r/q/I631f0d0bf20aaa7d0a3f9249bf3df755e0592270
[1] https://docs.opensearch.org/latest/install-and-configure/plugins/#install-a-plugin-from-a-zip-file

@dcausse has provided a organized list of plugins here

@colewhite Thanks for sharing that, we will definitely take a look. It's unfortunate that a distributed datastore doesn't have a distributed way to manage its plugins. We're also considering some other approaches as well now that we have working OpenSearch OCI images. I'm going to experiment with podman/quadlet - if that works, we could conditionally expose the plugins via bind mounts.

I stood up a 3-node OpenSearch 2 cluster in WMCS today to check if it were possible to remove the plugins from a running cluster without using rm commands. I'm using the query-insights plugin, which comes bundled with OpenSearch, as an example. From opensearch-1 , I ran the following commands:

curl -s http://0:9200/_cat/plugins | grep query-insights # check for plugin
opensearch-1.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0
opensearch-0.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0

export PATH=${PATH}:/usr/share/opensearch/bin # get OpenSearch binaries into your path
export OPENSEARCH_PATH_CONF=/etc/opensearch # point to the correct opensearch config
root@opensearch-1:~# opensearch-plugin remove query-insights
-> removing [query-insights]...

 ls -lhtra /usr/share/opensearch/plugins/ | grep query # the plugin directory is removed

 curl -s http://0:9200/_cat/plugins | grep query-insights # the plugin is still registered in OpenSearch
opensearch-1.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0
opensearch-0.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0

root@opensearch-1:~# curl -s http://0:9200/_cat/plugins | grep query-insights # the plugin is removed from the list
opensearch-0.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0

root@opensearch-1:~# apt-get install --reinstall opensearch
root@opensearch-1:~# ls -lhtra /usr/share/opensearch/plugins/ | grep query
drwxr-xr-x  2 root root 4.0K Apr 16 22:06 query-insights # the plugin folder is back

curl -s http://0:9200/_cat/plugins | grep query-insights # reinstalling the package also restarted the service, so the plugin is back
opensearch-0.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0
opensearch-1.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud query-insights                       2.19.5.0

So we can do this a bit more safely. I'm also going to look at systemd's PrivateMounts feature, it's possible we could keep all the plugins and only mount a subset into each service's namespace

Good news, it does look like PrivateMounts will work.

Given a systemd override of:

[Service]
# Give OpenSearch its own mount namespace
PrivateMounts=yes

# Replace plugins dir with an empty tmpfs
TemporaryFileSystem=/usr/share/opensearch/plugins:rw

BindPaths=
{%for plugin in allowed_plugins %}
BindPaths=/usr/share/opensearch/plugins/{{ plugin }}
{% endfor %}

And a list of allowed plugins:

allowed_plugins:
  - opensearch-job-scheduler
  - opensearch-knn
  - opensearch-ltr
  - opensearch-ml
  - opensearch-neural-search

It looks like we can filter out the unwanted plugins:

curl -s http://0:9200/_cat/plugins
opensearch-0.search.eqiad1.wikimedia.cloud opensearch-job-scheduler 2.19.5.0
opensearch-0.search.eqiad1.wikimedia.cloud opensearch-knn           2.19.5.0
opensearch-0.search.eqiad1.wikimedia.cloud opensearch-ltr           2.19.5.0
opensearch-0.search.eqiad1.wikimedia.cloud opensearch-ml            2.19.5.0
opensearch-0.search.eqiad1.wikimedia.cloud opensearch-neural-search 2.19.5.0
opensearch-1.search.eqiad1.wikimedia.cloud opensearch-job-scheduler 2.19.5.0
opensearch-1.search.eqiad1.wikimedia.cloud opensearch-knn           2.19.5.0
opensearch-1.search.eqiad1.wikimedia.cloud opensearch-ltr           2.19.5.0
opensearch-1.search.eqiad1.wikimedia.cloud opensearch-ml            2.19.5.0
opensearch-1.search.eqiad1.wikimedia.cloud opensearch-neural-search 2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud opensearch-job-scheduler 2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud opensearch-knn           2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud opensearch-ltr           2.19.5.0
opensearch-2.search.eqiad1.wikimedia.cloud opensearch-ml            2.19.5.0

While they remain on the disk:

root@opensearch-0:/usr/share/opensearch/plugins# ls
opensearch-alerting                   opensearch-job-scheduler       opensearch-performance-analyzer
opensearch-anomaly-detection          opensearch-knn                 opensearch-reports-scheduler
opensearch-asynchronous-search        opensearch-ltr                 opensearch-security
opensearch-cross-cluster-replication  opensearch-ml                  opensearch-security-analytics
opensearch-custom-codecs              opensearch-neural-search       opensearch-skills
opensearch-flow-framework             opensearch-notifications       opensearch-sql
opensearch-geospatial                 opensearch-notifications-core  opensearch-system-templates
opensearch-index-management           opensearch-observability

@bking the above approach seems really elegant (not quoting so I don't spam the ticket with a duplicate wall of text). this works around lots of the ugliness of some of the other hacks I/we've been putting in place to work around this. I'm fully in favor of this route.

Change #1273887 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] OpenSearch: Control which plugins we use via systemd PrivateMounts

https://gerrit.wikimedia.org/r/1273887

bking changed the status of subtask T423523: Handle typos/possibly update opensearch-analysis-stconvert plugin from Open to In Progress.Fri, Apr 17, 5:42 PM

bking closed subtask T423523: Handle typos/possibly update opensearch-analysis-stconvert plugin as Resolved.

Change #1273887 merged by Bking:

[operations/puppet@production] OpenSearch: Control which plugins we use via systemd PrivateMounts

https://gerrit.wikimedia.org/r/1273887

Change #1273937 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] opensearch: move var up so we can use it earlier

https://gerrit.wikimedia.org/r/1273937

Change #1273937 merged by Bking:

[operations/puppet@production] opensearch: move var up so we can use it earlier

https://gerrit.wikimedia.org/r/1273937

Change #1273943 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic1012: remove the deliberately-introduced typo

https://gerrit.wikimedia.org/r/1273943

Change #1273943 merged by Bking:

[operations/puppet@production] cloudelastic1012: remove the deliberately-introduced typo

https://gerrit.wikimedia.org/r/1273943

Change #1274061 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cloudelastic1012: override common_settings merge to first

https://gerrit.wikimedia.org/r/1274061

Change #1274075 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):