Page MenuHomePhabricator
Create Task
Maniphest T403301

Discuss OpenSearch 3 roadmap/future improvements
Closed, ResolvedPublic
Actions

Description

For context: OpenSearch 3 was released in May 2025.

The OpenSearch team provides Debian packages, but we do not currently use them†.
If we wished to mirror these upstream packages to our own APT repository, we would:

By now, OpenSearch is also used for a number of other use cases in production:

There is another use production use-case currently in development:

This new opensearch cluster will be hosted on the dse-k8s clusters and provisioned using the opensearch-k8s-operator. Block storage for the opensearch cluster pods will be provided by Ceph. Please see: T362105 for more details about this project.

Additionally, as the OpenSearch on K8s project develops, we've seen developer interest in running newer OpenSearch versions.

Creating this ticket to:

  • Discuss OpenSearch 3 roadmap w/Observability and agree on a plan
  • Discuss OpenSearch package ownership more generally. Potential questions:
    • Should we (observability , Discovery-Search , and Data-Platform-SRE ) share responsibility for building/publishing packages?
    • Would we benefit from using the upstream opensearch packages for some or all of the use-cases?
    • Is sharing a repo the best approach, or should we each have our own repos?
    • Could the build process be automated, perhaps with CI and/or an ansible playbook?

†Per @cwhite, we host their built deb artifacts in our (WMF) apt repo at thirdparty/opensearch2. The currently-available version from the WMF deb repo is 2.7.0, which was released in May 2023 .

Related Objects
Search...

Event Timeline

bking created this task.Aug 29 2025, 8:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 29 2025, 8:39 PM
bking added subscribers: colewhite, BTullis, RKemper, EBernhardson.Aug 29 2025, 8:39 PM
dr0ptp4kt subscribed.Sep 2 2025, 4:01 PM
Gehel added a project: observability.Sep 9 2025, 2:07 PM
hnowlan subscribed.Sep 10 2025, 10:51 AM
BTullis updated the task description. (Show Details)Sep 19 2025, 11:02 AM
BTullis assigned this task to bking.Sep 19 2025, 3:25 PM
BTullis triaged this task as Medium priority.
BTullis edited projects, added: Data-Platform-SRE (2025.09.05 - 2025.09.26); removed: Data-Platform-SRE.
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.

There is a useful Slack thread about this here.

bking updated the task description. (Show Details)Sep 19 2025, 4:38 PM
bking added a comment.Sep 19 2025, 4:45 PM

As mentioned in the above Slack thread, observability requires the ability to pick a specific OpenSearch version, as

Installing different versions between beta and the production datacenters for testing is a common pattern for us.

To verify this is an option, I've spun up trixie.search.eqiad.wikimedia.cloud in WMCS and followed the OpenSearch docs for installing version 2.x from a Debian repo .

After following the directions, I can confirm that the repo hosts 2.x versions back to 2.5.0:

root@trixie:~#  apt list -a opensearch
opensearch/stable 2.19.3 amd64
opensearch/stable 2.19.2 amd64
opensearch/stable 2.19.1 amd64
opensearch/stable 2.19.0 amd64
opensearch/stable 2.18.0 amd64
opensearch/stable 2.17.1 amd64
opensearch/stable 2.17.0 amd64
opensearch/stable 2.16.0 amd64
opensearch/stable 2.15.0 amd64
opensearch/stable 2.14.0 amd64
opensearch/stable 2.13.0 amd64
opensearch/stable 2.12.0 amd64
opensearch/stable 2.11.1 amd64
opensearch/stable 2.11.0 amd64
opensearch/stable 2.10.0 amd64
opensearch/stable 2.9.0 amd64
opensearch/stable 2.8.0 amd64
opensearch/stable 2.7.0 amd64
opensearch/stable 2.6.0 amd64
opensearch/stable 2.5.0 amd64

However, I also saw an error seemingly related to package verification:

W: https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt/dists/stable/InRelease: Policy will reject signature within a year, see --audit for de
tails

I'm not sure if this is a Trixie-specific thing. I'll try the same steps on Bookworm and do some more research.

bking added a comment.EditedSep 19 2025, 4:59 PM

This appears to be caused by Debian Trixie's upcoming rejection of keys containing an SHA1-based signature:

root@trixie:/etc/apt/sources.list.d# apt-get update --audit
W: https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt/dists/stable/InRelease: Policy will reject signature within a year, see --audit for details
A: https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt/dists/stable/InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is:
   Signing key on C5B7498965EFD1C2924BA9D539D319879310D3FC is not bound:
              No binding signature at time 2025-07-22T18:48:28Z
     because: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance
     because: SHA1 is not considered secure since 2026-02-01T00:00:00Z

Based on my search results, it seems many repos need to update their signing keys. I don't think this will be a blocker for us, but let me check on Bookworm as I promised earlier.

bking added a comment.Sep 19 2025, 5:28 PM

Update: I installed on a Bookworm host using the same process, and I can confirm that there are no warnings. I can also confirm that we can install the current version of OpenSearch 2.x that we're using in our repos (2.7.0):

bking@os-bookworm:~$ sudo apt-get install opensearch=2.7.0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  opensearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 748 MB of archives.
After this operation, 976 MB of additional disk space will be used.
Get:1 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt sta
ble/main amd64 opensearch amd64 2.7.0 [748 MB]
Fetched 748 MB in 24s (30.9 MB/s)
Selecting previously unselected package opensearch.
(Reading database ... 54733 files and directories currently installed.)
Preparing to unpack .../opensearch_2.7.0_amd64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.7.0) ...
Setting up opensearch (2.7.0) ...
Gehel edited projects, added: Data-Platform-SRE (2025.09.26 - 2025.10.17); removed: Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
BTullis added a comment.Sep 29 2025, 10:14 PM

Are we agreed then that we should do the following?

  • Update puppet to ensure that opensearch is pinned to a specific wherever it is installed
  • Enable mirroring of the upstream opensearch repositories (major versions 1 to 3)
  • Update the documentation to make it clear that we no longer build custom opensearch packages, but use the upstream versions

If we're happy with that, then I think we could probably make child tickets for those and call this one done.

BTullis mentioned this in T386730: Create thirdparty/opensearch13 repo for bullseye and remove thirdparty/opensearch1 repo.Sep 29 2025, 10:21 PM
BTullis added a subtask: T386730: Create thirdparty/opensearch13 repo for bullseye and remove thirdparty/opensearch1 repo.
bking mentioned this in T405985: OpenSearch on K8s: migrate from 2.7.0 to 2.8.0 or later version of the chart.Sep 29 2025, 10:33 PM
bking added a comment.Sep 30 2025, 5:02 PM

@BTullis I'm happy with the above, @colewhite any objections from the observability side?

I'd suggest mirroring the OpenSearch 2 repo last and only after we get the OK from Observability.

hnowlan moved this task from Inbox to Radar on the observability board.Oct 1 2025, 2:48 PM
colewhite added a comment.Oct 1 2025, 10:53 PM
In T403301#11227254, @BTullis wrote:

Are we agreed then that we should do the following?

  • Update puppet to ensure that opensearch is pinned to a specific wherever it is installed
  • Enable mirroring of the upstream opensearch repositories (major versions 1 to 3)
  • Update the documentation to make it clear that we no longer build custom opensearch packages, but use the upstream versions
In T403301#11230382, @bking wrote:

@colewhite any objections from the observability side?

The plan looks good to me. We have bandwidth to handle possible fallout and have ways to test the changes.

Gehel added a project: Essential-Work.Oct 7 2025, 8:51 AM
bking closed subtask T386730: Create thirdparty/opensearch13 repo for bullseye and remove thirdparty/opensearch1 repo as Declined.Oct 13 2025, 1:57 PM
bking mentioned this in T407123: Mirror OpenSearch repos from upstream.Oct 13 2025, 2:06 PM
BTullis mentioned this in T407199: Pin opensearch and logstash related package versions in puppet to avoid updates when we mirror the upstream repositories.Oct 14 2025, 10:16 AM
bking closed this task as Resolved.Oct 14 2025, 2:06 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.

Since we're all in agreement on the plan, I'm going to close this one out. Feel free to reopen if I missed something.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 17 2025, 8:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits