Page MenuHomePhabricator
Create Task
Maniphest T407199

Pin opensearch and logstash related package versions in puppet to avoid updates when we mirror the upstream repositories
Closed, ResolvedPublic
Actions

Description

We currently have several roles where the opensearch, opensearch-dashboards, logstash-oss, and logstash-plugins packages are installed.

In order to support a greater range of OpenSearch cluster versions, particularly under Kubernetes, we are planning to implement T407123: Mirror OpenSearch repos from upstream.
This will bring about a change to the contents of the thirdparty/opensearch2repository behaves.

Specifically, our current version of this repository only has one version of each of these packages.
For example, here is the current state of thirdparty/opensearch2 for the bookworm-wikimedia distribution.

btullis@apt1002:~$ sudo -i reprepro -C thirdparty/opensearch2 list bookworm-wikimedia
bookworm-wikimedia|thirdparty/opensearch2|amd64: elasticsearch-curator 5.8.5-1~wmf5+deb12u1
bookworm-wikimedia|thirdparty/opensearch2|amd64: logstash-oss 1:7.16.3-1
bookworm-wikimedia|thirdparty/opensearch2|amd64: logstash-plugins 7.16.3-1
bookworm-wikimedia|thirdparty/opensearch2|amd64: opensearch 2.7.0
bookworm-wikimedia|thirdparty/opensearch2|amd64: opensearch-dashboards 2.7.0
bookworm-wikimedia|thirdparty/opensearch2|i386: logstash-plugins 7.16.3-1

However, as shown during our investigation in T403301#11197942, the upstream repositories contain many versions of each of these packages.

Therefore, in order to prevent unwanted package upgrades after syning the repositories, we need to pin the versions of opensearch and logstash related packages before syncing the upstream opensearch2 repositories.

Note that elasticsearch-curator will also need to be moved to its own apt component, since this is not present in the upstream opensearch repositories.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Pin the logstash and logstash-plugins everywhere they are installedBtullisoperations/puppetproduction+189 -215
Pin the version of opensearch-dashboards wherever it is usedBtullisoperations/puppetproduction+54 -54
Pin the version of opensearch wherever it is installedBtullisoperations/puppetproduction+60 -67
Update the wikimedia-opensearch apt repository signed-by for bullseyeBtullisoperations/puppetproduction+13 -0
Change the component from where we install elasticsearch-curatorBtullisoperations/puppetproduction+9 -5
Update the apt components used for elasticsearch-curatorBtullisoperations/puppetproduction+2 -15
aptrepo: add component/curator5 to bullseye/bookwormCwhiteoperations/puppetproduction+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.Oct 14 2025, 10:16 AM
BTullis triaged this task as High priority.
gerritbot added a comment.Oct 14 2025, 11:42 AM

Change #1196022 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Pin the version of opensearch wherever it is installed

https://gerrit.wikimedia.org/r/1196022

gerritbot added a project: Patch-For-Review.Oct 14 2025, 11:42 AM

Change #1196023 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Pin the version of opensearch-dashboards wherever it is used

https://gerrit.wikimedia.org/r/1196023

gerritbot added a comment.Oct 14 2025, 12:43 PM

Change #1196057 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Pin the logstash and logstash-plugins everywhere they are installed

https://gerrit.wikimedia.org/r/1196057

BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 14 2025, 3:57 PM
Gehel edited projects, added: Data-Platform-SRE (2025.10.17 - 2025.11.07); removed: Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:49 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.
Gehel added a project: OKR-Work.Oct 17 2025, 1:47 PM
BTullis moved this task from In Progress to Needs Review on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.Oct 17 2025, 3:38 PM

I think that these three patches are all ready for a review now.

BTullis added a comment.Oct 17 2025, 3:55 PM

Also, one more thing occurs to me, which is that we need to change the way that we install elasticsearch-curator.
At the moment, we install a verison that we have copied to the thirdparty/opensearch1 and thirdparty/opensearch2 repos.

btullis@cumin1003:~$ sudo cumin 'R:Package = elasticsearch-curator' 'apt-cache policy elasticsearch-curator'
30 hosts will be targeted:
apifeatureusage2001.codfw.wmnet,apifeatureusage1001.eqiad.wmnet,logging-hd[2001-2005].codfw.wmnet,logging-hd[1001-1005].eqiad.wmnet,logging-sd[2001-2004].codfw.wmnet,logging-sd[1001-1004].eqiad.wmnet,logstash[2033-2037].codfw.wmnet,logstash[1033-1037].eqiad.wmnet
OK to proceed on 30 hosts? Enter the number of affected hosts to confirm or "q" to quit: 30
===== NODE GROUP =====                                                                                                                                                                                             
(28) logging-hd[2001-2005].codfw.wmnet,logging-hd[1001-1005].eqiad.wmnet,logging-sd[2001-2004].codfw.wmnet,logging-sd[1001-1004].eqiad.wmnet,logstash[2033-2037].codfw.wmnet,logstash[1033-1037].eqiad.wmnet       
----- OUTPUT of 'apt-cache policy...icsearch-curator' -----                                                                                                                                                        
elasticsearch-curator:                                                                                                                                                                                             
  Installed: 5.8.5-1~wmf5+deb12u1                                                                                                                                                                                  
  Candidate: 5.8.5-1~wmf5+deb12u1
  Version table:
 *** 5.8.5-1~wmf5+deb12u1 1001
       1001 http://apt.wikimedia.org/wikimedia bookworm-wikimedia/thirdparty/opensearch2 amd64 Packages
        100 /var/lib/dpkg/status
     5.8.1-4.1 500
        500 http://mirrors.wikimedia.org/debian bookworm/main amd64 Packages
===== NODE GROUP =====                                                                                                                                                                                             
(2) apifeatureusage2001.codfw.wmnet,apifeatureusage1001.eqiad.wmnet                                                                                                                                                
----- OUTPUT of 'apt-cache policy...icsearch-curator' -----                                                                                                                                                        
elasticsearch-curator:                                                                                                                                                                                             
  Installed: 5.8.5-1~wmf4+deb11u1                                                                                                                                                                                  
  Candidate: 5.8.5-1~wmf5+deb11u1
  Version table:
     5.8.5-1~wmf5+deb11u1 1001
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/thirdparty/opensearch1 amd64 Packages
 *** 5.8.5-1~wmf4+deb11u1 100
        100 /var/lib/dpkg/status
     5.8.1-1 500
        500 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages
================                                                                                                                                                                                                   
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (30/30) [00:02<00:00, 12.31hosts/s]
FAIL |                                                                                                                                                                            |   0% (0/30) [00:02<?, ?hosts/s]
100.0% (30/30) success ratio (>= 100.0% threshold) for command: 'apt-cache policy...icsearch-curator'.
100.0% (30/30) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
btullis@cumin1003:~$

I think that we can use the existing thirdparty/curator5 component, but just copy the updated deb into there:

btullis@apt1002:~$ sudo -i reprepro -C thirdparty/elasticsearch-curator5 list bookworm-wikimedia
bookworm-wikimedia|thirdparty/elasticsearch-curator5|amd64: elasticsearch-curator 5.6.0

btullis@apt1002:~$ sudo -i reprepro -C thirdparty/elasticsearch-curator5 list bullseye-wikimedia
bullseye-wikimedia|thirdparty/elasticsearch-curator5|amd64: elasticsearch-curator 5.6.0
BTullis added a comment.Oct 17 2025, 4:01 PM

I have updated the deb in these components.

btullis@apt1002:~$ sudo -i reprepro -C thirdparty/elasticsearch-curator5 includedeb bullseye-wikimedia /srv/wikimedia/pool/thirdparty/opensearch2/e/elasticsearch-curator/elasticsearch-curator_5.8.5-1~wmf5+deb11u1_amd64.deb
Exporting indices...
btullis@apt1002:~$ sudo -i reprepro -C thirdparty/elasticsearch-curator5 includedeb bookworm-wikimedia /srv/wikimedia/pool/thirdparty/opensearch2/e/elasticsearch-curator/elasticsearch-curator_5.8.5-1~wmf5+deb12u1_amd64.deb
Exporting indices...
Deleting files no longer referenced...

btullis@apt1002:~$ sudo -i reprepro -C thirdparty/elasticsearch-curator5 list bullseye-wikimedia
bullseye-wikimedia|thirdparty/elasticsearch-curator5|amd64: elasticsearch-curator 5.8.5-1~wmf5+deb11u1

btullis@apt1002:~$ sudo -i reprepro -C thirdparty/elasticsearch-curator5 list bookworm-wikimedia
bookworm-wikimedia|thirdparty/elasticsearch-curator5|amd64: elasticsearch-curator 5.8.5-1~wmf5+deb12u1
gerritbot added a comment.Oct 17 2025, 4:13 PM

Change #1196942 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Change the component from where we install elasticsearch-curator

https://gerrit.wikimedia.org/r/1196942

BTullis updated the task description. (Show Details)Oct 17 2025, 4:15 PM
gerritbot added a comment.Oct 23 2025, 9:26 AM

Change #1198287 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a new apt component for our custom elasticsearch-curator build

https://gerrit.wikimedia.org/r/1198287

BTullis renamed this task from Pin opensearch and logstash package versions in puppet to avoid updates when we mirror the upstream repositories to Pin opensearch and logstash related package versions in puppet to avoid updates when we mirror the upstream repositories.Oct 23 2025, 11:12 AM
BTullis updated the task description. (Show Details)
gerritbot added a comment.Oct 23 2025, 8:12 PM

Change #1198392 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] aptrepo: add component/curator5 to bullseye/bookworm

https://gerrit.wikimedia.org/r/1198392

gerritbot added a comment.Oct 23 2025, 8:16 PM

Change #1198392 abandoned by Cwhite:

[operations/puppet@production] aptrepo: add component/curator5 to bullseye/bookworm

Reason:

in favor of a5bf9c590f6d732da5346886add6a21d303c0183

https://gerrit.wikimedia.org/r/1198392

gerritbot added a comment.Nov 5 2025, 12:40 PM

Change #1198287 merged by Btullis:

[operations/puppet@production] Update the apt components used for elasticsearch-curator

https://gerrit.wikimedia.org/r/1198287

BTullis added a comment.Nov 5 2025, 2:17 PM

I have removed the old thirdpaty/elasticsearch-curator5 component like this:

btullis@apt1002:~$ sudo -i reprepro --delete clearvanished
Deleting vanished identifier 'bookworm-wikimedia|thirdparty/elasticsearch-curator5|amd64'.
Deleting vanished identifier 'bullseye-wikimedia|thirdparty/elasticsearch-curator5|amd64'.
Deleting files no longer referenced...

I have also copied the curator binaries to the new component/curator5 component like this:

btullis@apt1002:~$ sudo -i reprepro -C component/curator5 includedeb bullseye-wikimedia /srv/wikimedia/pool/thirdparty/opensearch2/e/elasticsearch-curator/elasticsearch-curator_5.8.5-1~wmf5+deb11u1_amd64.deb
Exporting indices...
btullis@apt1002:~$ sudo -i reprepro -C component/curator5 includedeb bookworm-wikimedia /srv/wikimedia/pool/thirdparty/opensearch2/e/elasticsearch-curator/elasticsearch-curator_5.8.5-1~wmf5+deb12u1_amd64.deb
Exporting indices...

Now these show up correctly:

btullis@apt1002:~$ sudo -i reprepro -C component/curator5 ls elasticsearch-curator
elasticsearch-curator | 5.8.5-1~wmf5+deb11u1 | bullseye-wikimedia | amd64
elasticsearch-curator | 5.8.5-1~wmf5+deb12u1 | bookworm-wikimedia | amd64

I can proceed to deploy this, now: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1196942

gerritbot added a comment.Nov 5 2025, 3:27 PM

Change #1196942 merged by Btullis:

[operations/puppet@production] Change the component from where we install elasticsearch-curator

https://gerrit.wikimedia.org/r/1196942

gerritbot added a comment.Nov 5 2025, 3:50 PM

Change #1202183 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the wikimedia-opensearch apt repository signed-by for bullseye

https://gerrit.wikimedia.org/r/1202183

gerritbot added a comment.Nov 5 2025, 3:53 PM

Change #1202183 merged by Btullis:

[operations/puppet@production] Update the wikimedia-opensearch apt repository signed-by for bullseye

https://gerrit.wikimedia.org/r/1202183

gerritbot added a comment.Nov 5 2025, 6:28 PM

Change #1196022 merged by Btullis:

[operations/puppet@production] Pin the version of opensearch wherever it is installed

https://gerrit.wikimedia.org/r/1196022

gerritbot added a comment.Nov 5 2025, 6:53 PM

Change #1196023 merged by Btullis:

[operations/puppet@production] Pin the version of opensearch-dashboards wherever it is used

https://gerrit.wikimedia.org/r/1196023

Gehel edited projects, added: Data-Platform-SRE (2025.11.07 - 2025.11.28); removed: Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:30 PM
Gehel moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
gerritbot added a comment.Nov 10 2025, 10:57 PM

Change #1196057 merged by Btullis:

[operations/puppet@production] Pin the logstash and logstash-plugins everywhere they are installed

https://gerrit.wikimedia.org/r/1196057

BTullis closed this task as Resolved.Nov 10 2025, 11:03 PM
BTullis moved this task from Needs Review to Done on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
Maintenance_bot removed a project: Patch-For-Review.Nov 10 2025, 11:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.Nov 14 2025, 10:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits