Per parent ticket (and the discussion in T403301) , we'll need to create the
thirdparty/opensearch2 thirdparty/opensearch3
repos for Bullseye, Bookworm, and Trixie (Trixie might be blocked; we'll need to check upstream's signing key. More details in this comment ).
The combination of OpenSearch 3/Bookworm is the highest priority.
AC: The following repos are mirrored to from upstream OpenSearch to WMF's internal repo infra.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T362105 EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6 | |||
| Resolved | bking | T397246 Deploy OpenSearch in dse-k8s | |||
| Resolved | bking | T404907 Deploy a test opensearch cluster on DSE k8s, using opensearch operator | |||
| Open | None | T405985 OpenSearch on K8s: migrate from 2.7.0 to 2.8.0 or later version of the chart | |||
| Resolved | bking | T407123 Mirror OpenSearch repos from upstream | |||
| Open | None | T407126 OpenSearch on K8s: build OpenSearch images for latest versions 2x and 3x | |||
| Resolved | BTullis | T407199 Pin opensearch and logstash related package versions in puppet to avoid updates when we mirror the upstream repositories | |||
| Resolved | colewhite | T410795 Logstash: No longer able to install OpenSearch 2.7.0 from internal apt repo |
Change #1196949 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Configure reprepro to mirror upstream opensearch2 and opensearch3 repos
Change #1196949 merged by Bking:
[operations/puppet@production] Configure reprepro to mirror upstream opensearch2 and opensearch3 repos
Change #1204639 had a related patch set uploaded (by Bking; author: Btullis):
[operations/puppet@production] Configure reprepro to mirror upstream opensearch2 and opensearch3 repos
Change #1204639 merged by Bking:
[operations/puppet@production] Configure reprepro to mirror upstream opensearch2 and opensearch3 repos
Change #1207195 had a related patch set uploaded (by Bking; author: DCausse):
[operations/puppet@production] apt: update opensearch3 key
Change #1207195 merged by Bking:
[operations/puppet@production] apt: update opensearch3 key
Change #1208020 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] opensearch: Fix typo in repo GPG key filename.
Change #1208020 merged by Bking:
[operations/puppet@production] opensearch: Fix typo in repo GPG key filename.
I took another look at this today, as we (DPE SRE) are starting to build our first Docker images from the upstream package. Suddenly, I realize that I don't know enough about how our repo mirroring works. Let me explain:
If I configure the upstream OpenSearch 2.x repo , I can see many different versions of the 2.x release:
apt policy opensearch
opensearch:
Installed: (none)
Candidate: 2.19.4
Version table:
2.19.4 500
500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
2.19.3 500
500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
2.19.2 500
500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
2.19.1 500
500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
2.19.0 500
500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages
2.18.0 500
500 https://artifacts.opensearch.org/releases/bundle/
[...]
2.5.0 500
500 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 PackagesBut if I use our mirror, I only see the latest available version:
apt policy opensearch
opensearch:
Installed: (none)
Candidate: 2.19.4
Version table:
2.19.4 1001
1001 http://apt.wikimedia.org/wikimedia bookworm-wikimedia/thirdparty/opensearch2 amd64 PackagesWe would prefer to have older versions of the package in the repo as well, if possible. Maybe I missed a setting or something when I set up a mirror? I'll go back and consult the docs, and I've also pinged in #wikimedia-sre-foundations IRC.
Checking the docs, it seems that reprepro doesn't support hosting multiple versions of the same package in the repo:
If you need to keep multiple versions of the same package in the repo, you will need to create versioned components, as reprepro doesn't support keeping foo_10-1.deb and foo_20-1.deb at the same time in the same component.
@Muehlenhoff can you confirm that there is no supported way to host multiple versions of the same package in the same repo at the moment?
Depends on what you plan to do:
You can't have opensearch_1.0.deb, opensearch_2.0 and opensearch_3.0 in the same component, but it's possible to have to opensearch_1.0.deb in component/opensearch1, opensearch_2.0 in component/opensearch2 and opensearch_3.0 in component/opensearch3. We're already following this scheme for a lot of components like Elastic already, so maybe I misunderstand your question?
Closing the loop for IRC conversation with @Muehlenhoff :
I tried copying to Bullseye today, and it looks like someone already beat me to it:
sudo -i reprepro copy bullseye-wikimedia bookworm-wikimedia opensearch Warning: replacing 'opensearch' version '2.19.4' with equal version '2.19.4' in 'bullseye-wikimedia|thirdparty/opensearch2|amd64'! Exporting indices...
This means the AC is completed. Closing...