Page MenuHomePhabricator
Create Task
Maniphest T422860

Migrate Cloudelastic to OpenSearch 2.x
Closed, ResolvedPublic
Actions

Description

Since we can't use our typical test cluster (relforge) as it's occupied by a Semantic Search experiment (ref T413969 ), we will start our OpenSearch 2.x migration on the cloudelastic cluster. Creating this ticket to:

  • Update the cluster from OpenSearch 1.x->2.x
  • Document any lessons learned: Added docs on our use of systemd for plugin management

Details

Other Assignee
RKemper
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cloudelastic: get rid of merge overridesoperations/puppetproduction+0 -4
cloudelastic: remove a plugin path that doesn't existoperations/puppetproduction+4 -2
cloudelastic: Add back the opensearch-ltr pluginoperations/puppetproduction+2 -0
cloudelastic: Another ugly plugins patchoperations/puppetproduction+22 -80
cloudelastic: Another required plugins update patchoperations/puppetproduction+2 -0
cloudelastic: update allowed plugins listoperations/puppetproduction+2 -2
cloudelastic: set role-level hiera for OpenSearch 2/Trixieoperations/puppetproduction+6 -6
cloudelastic: add missing extra-analysis pluginsoperations/puppetproduction+15 -0
opensearch 1: parameterize disk threshold values/up limits in ceoperations/puppetproduction+16 -8
cloudelastic: prepare cloudelastic1011 for Trixie/OpenSearch 2operations/puppetproduction+56 -1
prometheus: fix wmf-elasticsearch-exporter listen address on Trixieoperations/puppetproduction+2 -1
cloudelastic1012: move back to production roleoperations/puppetproduction+1 -5
cloudelastic1012: move back to insetupoperations/puppetproduction+5 -1
Cirrussearch: remove unused hiera filesoperations/puppetproduction+0 -297
cloudelastic1012: Set LVS config for opensearch_2operations/puppetproduction+13 -0
cloudelastic1012: full common_settings override for OS2operations/puppetproduction+11 -4
cloudelastic1012: full common_settings override for OS2operations/puppetproduction+17 -2
cloudelastic1012: full common_settings override for OS2operations/puppetproduction+17 -2
cloudelastic1012: remove the deliberately-introduced typooperations/puppetproduction+1 -1
opensearch: move var up so we can use it earlieroperations/puppetproduction+4 -2
OpenSearch: Control which plugins we use via systemd PrivateMountsoperations/puppetproduction+29 -0
opensearch: strip bundled plugins before WMF pkgoperations/puppetproduction+25 -0
opensearch: allowlist upstream plugins + overwriteoperations/puppetproduction+54 -13
cloudelastic: temporarily add "working typos" for pluginsoperations/puppetproduction+17 -0
cloudelastic: fix java path typooperations/puppetproduction+1 -1
opensearch: correct o11y usage in commentoperations/puppetproduction+1 -1
opensearch: hack around upstream 2.x+ packagesoperations/puppetproduction+20 -0
nginx tls proxy: remove defunct directiveoperations/puppetproduction+0 -1
cloudelastic: remove logstash profileoperations/puppetproduction+0 -1
opensearch: move cloudelastic1012 back into prod roleoperations/puppetproduction+1 -5
cirrussearch: move cloudelastic1012 to insetupoperations/puppetproduction+6 -2
cloudelastic: Prepare for opensearch 2operations/puppetproduction+32 -4
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Tue, Apr 14, 1:25 PM

Change #1270511 merged by Bking:

[operations/puppet@production] opensearch: hack around upstream 2.x+ packages

https://gerrit.wikimedia.org/r/1270511

gerritbot added a comment.Tue, Apr 14, 2:15 PM

Change #1270953 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch: correct o11y usage in comment

https://gerrit.wikimedia.org/r/1270953

bking added a subtask: T423291: Build new wmf-opensearch-search-plugins package for opensearch 2.x/trixie and ensure we don't install/enable any unwanted plugins in prod.Tue, Apr 14, 2:41 PM
gerritbot added a comment.Tue, Apr 14, 4:16 PM

Change #1270953 merged by Cwhite:

[operations/puppet@production] opensearch: correct o11y usage in comment

https://gerrit.wikimedia.org/r/1270953

bking added a subtask: T423327: Explore options for OpenSearch 2.x/3.x plugin packaging and distribution.Tue, Apr 14, 6:22 PM
gerritbot added a comment.Wed, Apr 15, 7:02 AM

Change #1271473 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] opensearch: strip bundled plugins before WMF pkg

https://gerrit.wikimedia.org/r/1271473

MoritzMuehlenhoff subscribed.Wed, Apr 15, 7:19 AM

@bking Wrt the issues with the broken import of the madvise package; I went ahead and rebuilt it as 0.2+deb13u1. While the only dependency of that package is in glibc with a stable ABI, it's still preferable to rebuild it with GCC 15 from Trixie. The new version also resolves the versioning/import issue. I've synced the debs to my home on cloudelastic1012, but didn't install them yet since I didn't want to meddle with any ongoing tests of you. When the time is right, please install them on 1012 and if they are fine, I'll import them to apt.w.o.

bking added a comment.Wed, Apr 15, 3:10 PM

@MoritzMuehlenhoff, I've installed the packages as you requested and I can confirm they installed cleanly. Feel free to publish them to the repos.

Thanks for your help!

bking mentioned this in P90344 Reprepro error T422860.Wed, Apr 15, 3:12 PM
bking changed the status of subtask T423291: Build new wmf-opensearch-search-plugins package for opensearch 2.x/trixie and ensure we don't install/enable any unwanted plugins in prod from Duplicate to Resolved.
gerritbot added a comment.Wed, Apr 15, 3:19 PM

Change #1271473 merged by Bking:

[operations/puppet@production] opensearch: strip bundled plugins before WMF pkg

https://gerrit.wikimedia.org/r/1271473

CodeReviewBot added a comment.Wed, Apr 15, 4:42 PM

bking merged https://gitlab.wikimedia.org/repos/search-platform/opensearch-plugins-deb/-/merge_requests/20

Remove already-installed packages

gerritbot added a comment.Wed, Apr 15, 5:06 PM

Change #1271818 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: fix java path typo

https://gerrit.wikimedia.org/r/1271818

gerritbot added a comment.Wed, Apr 15, 5:13 PM

Change #1271818 merged by Bking:

[operations/puppet@production] cloudelastic: fix java path typo

https://gerrit.wikimedia.org/r/1271818

ops-monitoring-bot added a comment.Wed, Apr 15, 9:06 PM

Icinga downtime and Alertmanager silence (ID=396a17ce-b27d-41be-a6ce-921c607989da) set by bking@cumin2002 for 7 days, 0:00:00 on 1 host(s) and their services with reason: still fixing Puppet

cloudelastic1012.eqiad.wmnet
gerritbot added a comment.Wed, Apr 15, 9:21 PM

Change #1271929 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: temporarily add "working typos" for plugins

https://gerrit.wikimedia.org/r/1271929

gerritbot added a comment.Wed, Apr 15, 9:27 PM

Change #1271929 merged by Bking:

[operations/puppet@production] cloudelastic: temporarily add "working typos" for plugins

https://gerrit.wikimedia.org/r/1271929

gerritbot added a comment.Wed, Apr 15, 10:03 PM

Change #1271947 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] opensearch: allowlist upstream-only plugins

https://gerrit.wikimedia.org/r/1271947

Stashbot added a comment.Thu, Apr 16, 6:55 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-16T06:55:17Z] <moritzm> imported opensearch-madvise 0.2+deb13u1 to component/opensearch2 of trixie-wikimedia T422860

MoritzMuehlenhoff added a comment.Thu, Apr 16, 6:57 AM
In T422860#11824928, @bking wrote:

@MoritzMuehlenhoff, I've installed the packages as you requested and I can confirm they installed cleanly. Feel free to publish them to the repos.

Nice! I've imported the new package into component/opensearch2 for trixie-wikimedia

gerritbot added a comment.Fri, Apr 17, 5:03 PM

Change #1273887 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] OpenSearch: Control which plugins we use via systemd PrivateMounts

https://gerrit.wikimedia.org/r/1273887

gerritbot added a comment.Fri, Apr 17, 7:22 PM

Change #1273887 merged by Bking:

[operations/puppet@production] OpenSearch: Control which plugins we use via systemd PrivateMounts

https://gerrit.wikimedia.org/r/1273887

gerritbot added a comment.Fri, Apr 17, 7:37 PM

Change #1273937 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] opensearch: move var up so we can use it earlier

https://gerrit.wikimedia.org/r/1273937

gerritbot added a comment.Fri, Apr 17, 7:42 PM

Change #1273937 merged by Bking:

[operations/puppet@production] opensearch: move var up so we can use it earlier

https://gerrit.wikimedia.org/r/1273937

gerritbot added a comment.Fri, Apr 17, 7:55 PM

Change #1273943 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic1012: remove the deliberately-introduced typo

https://gerrit.wikimedia.org/r/1273943

gerritbot added a comment.Fri, Apr 17, 7:57 PM

Change #1273943 merged by Bking:

[operations/puppet@production] cloudelastic1012: remove the deliberately-introduced typo

https://gerrit.wikimedia.org/r/1273943

gerritbot added a comment.Fri, Apr 17, 9:48 PM

Change #1274061 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cloudelastic1012: override common_settings merge to first

https://gerrit.wikimedia.org/r/1274061

gerritbot added a comment.Fri, Apr 17, 9:55 PM

Change #1274075 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cloudelastic1012: full common_settings override for OS2

https://gerrit.wikimedia.org/r/1274075

gerritbot added a comment.Fri, Apr 17, 9:55 PM

Change #1274075 abandoned by Ryan Kemper:

[operations/puppet@production] cloudelastic1012: full common_settings override for OS2

Reason:

meant to update https://gerrit.wikimedia.org/r/c/operations/puppet/+/1274061; abandoning

https://gerrit.wikimedia.org/r/1274075

gerritbot added a comment.Fri, Apr 17, 10:00 PM

Change #1274061 merged by Ryan Kemper:

[operations/puppet@production] cloudelastic1012: full common_settings override for OS2

https://gerrit.wikimedia.org/r/1274061

gerritbot added a comment.Fri, Apr 17, 10:54 PM

Change #1274134 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cloudelastic1012: full common_settings override for OS2

https://gerrit.wikimedia.org/r/1274134

gerritbot added a comment.Fri, Apr 17, 10:57 PM

Change #1274134 merged by Ryan Kemper:

[operations/puppet@production] cloudelastic1012: full common_settings override for OS2

https://gerrit.wikimedia.org/r/1274134

gerritbot added a comment.Mon, Apr 20, 3:04 PM

Change #1275435 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic1012: Set LVS config for opensearch_2

https://gerrit.wikimedia.org/r/1275435

gerritbot added a comment.Mon, Apr 20, 3:09 PM

Change #1275435 merged by Bking:

[operations/puppet@production] cloudelastic1012: Set LVS config for opensearch_2

https://gerrit.wikimedia.org/r/1275435

gerritbot added a comment.Mon, Apr 20, 3:38 PM

Change #1275444 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] Cirrussearch: remove unused hiera files

https://gerrit.wikimedia.org/r/1275444

gerritbot added a comment.Mon, Apr 20, 3:41 PM

Change #1275444 merged by Bking:

[operations/puppet@production] Cirrussearch: remove unused hiera files

https://gerrit.wikimedia.org/r/1275444

gerritbot added a comment.Mon, Apr 20, 5:00 PM

Change #1275473 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic1012: move back to insetup

https://gerrit.wikimedia.org/r/1275473

gerritbot added a comment.Mon, Apr 20, 5:00 PM

Change #1275473 merged by Bking:

[operations/puppet@production] cloudelastic1012: move back to insetup

https://gerrit.wikimedia.org/r/1275473

ops-monitoring-bot added a comment.Mon, Apr 20, 5:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1012.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mon, Apr 20, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1012.eqiad.wmnet with OS trixie completed:

  • cloudelastic1012 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604201718_bking_545568_cloudelastic1012.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mon, Apr 20, 5:45 PM

Change #1275485 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic1012: move back to production role

https://gerrit.wikimedia.org/r/1275485

gerritbot added a comment.Mon, Apr 20, 5:46 PM

Change #1275485 merged by Bking:

[operations/puppet@production] cloudelastic1012: move back to production role

https://gerrit.wikimedia.org/r/1275485

bking added a comment.EditedMon, Apr 20, 7:36 PM

After reimaging cloudelastic2012, it is back up and ready for testing.

I've removed it from load balancer rotation, shut off Puppet, and have stopped all instances except psi (port 9600), which I'm using as our guinea pig.

So far, I've gotten the following errors:

Caused by: java.lang.IllegalStateException: index [.ltrstore/vCo9DZu5Qt-3QbtmBy1d7Q] version not supported: 6.5.4 minimum compatible index version is: 7.

This index is part of OpenSearch's machine learning/Learning to Rank feature set . It is not used in cloudelastic, but for production we may have to do something like*:

  • create a new named store
  • reload the data (maybe via reindex api, needs testing),

-repoint queries at the new feature store

  • get rid of the old one

The next error I've seen is very similar:

java.lang.IllegalStateException: index [mw_cirrus_metastore_1659365741/ugKwuXOpRjiti8dY67m9OA] version not supported: 6.8.23 minimum compatibl

Per codesearch , cirrussearch (the Mediawiki extension that provides OpenSearch support) uses the mw_cirrus_metastore index to store the state of administrative tasks. We're still working out a plan to upgrade this index gracefully as I write this.

*suggested by @EBernhardson in Wikimedia-Search IRC

bking added a subscriber: EBernhardson.Mon, Apr 20, 7:47 PM
bking added a comment.Mon, Apr 20, 9:01 PM

We had a few more indices to delete before the existing OpenSearch 1.x clusters would allow an OpenSearch 2 node to join. We can find the problem indices with this one-liner:

curl -s localhost:${PORT}/_all/_settings | jq -r 'to_entries[] | "\(.key) \(.value.settings.index.version.created)"' | grep -v 135249827
(135249827 means the index was created on OpenSearch 1, anything not matching that will be a problem).

The problem indices for Cloudelastic were:

  • .ltrstore as described above
  • mw_cirrus_metastore also described above
  • .tasks - used internally by OpenSearch to keep track of running tasks. Safe enough to delete in most circumstances (if you just lost a bunch of data and were waiting for OpenSearch to recreate shards, probably not).

We will have to be a bit more cautious for the production clusters, but I think just need a few reimages to get Cloudelastic onto OpenSearch 2.x.

gerritbot added a comment.Mon, Apr 20, 9:31 PM

Change #1275535 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] prometheus: fix wmf-elasticsearch-exporter listen address on Trixie

https://gerrit.wikimedia.org/r/1275535

gerritbot added a comment.Mon, Apr 20, 9:43 PM

Change #1275535 merged by Ryan Kemper:

[operations/puppet@production] prometheus: fix wmf-elasticsearch-exporter listen address on Trixie

https://gerrit.wikimedia.org/r/1275535

bking added a subscriber: RKemper.Mon, Apr 20, 10:01 PM

Note that we also ran into a problem with the prometheus exporter and Python 3.13, which comes with Trixie. @RKemper 's patch above fixes that.

bking mentioned this in T421757: ☂️ Migrate production OpenSearch clusters from 1.x-2.x ☂️.Mon, Apr 20, 10:11 PM
bking closed subtask T423327: Explore options for OpenSearch 2.x/3.x plugin packaging and distribution as Resolved.Tue, Apr 21, 10:35 PM
gerritbot added a comment.Thu, Apr 23, 9:32 PM

Change #1276804 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: prepare cloudelastic1011 for Trixie/OpenSearch 2

https://gerrit.wikimedia.org/r/1276804

gerritbot added a comment.Thu, Apr 23, 9:35 PM

Change #1276804 merged by Bking:

[operations/puppet@production] cloudelastic: prepare cloudelastic1011 for Trixie/OpenSearch 2

https://gerrit.wikimedia.org/r/1276804

ops-monitoring-bot added a comment.Thu, Apr 23, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1011.eqiad.wmnet with OS trixie

gerritbot added a comment.Thu, Apr 23, 10:04 PM

Change #1276818 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: set role-level hiera for OpenSearch 2/Trixie

https://gerrit.wikimedia.org/r/1276818

ops-monitoring-bot added a comment.Thu, Apr 23, 10:21 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1011.eqiad.wmnet with OS trixie completed:

  • cloudelastic1011 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604232154_bking_3545146_cloudelastic1011.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Gehel edited projects, added Data-Platform-SRE (2026-04-24 - 2026-05-15); removed Data-Platform-SRE (2026-03-27 - 2026-04-17).Fri, Apr 24, 9:16 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
ops-monitoring-bot added a comment.Fri, Apr 24, 3:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Fri, Apr 24, 3:35 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.eqiad.wmnet with OS trixie executed with errors:

  • cloudelastic1010 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudelastic1010.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Fri, Apr 24, 7:49 PM

Change #1277180 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] opensearch 1: parameterize disk threshold values

https://gerrit.wikimedia.org/r/1277180

gerritbot added a comment.Fri, Apr 24, 8:42 PM

Change #1277180 merged by Bking:

[operations/puppet@production] opensearch 1: parameterize disk threshold values/up limits in ce

https://gerrit.wikimedia.org/r/1277180

gerritbot added a comment.Fri, Apr 24, 9:33 PM

Change #1277194 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cloudelastic: add missing extra-analysis plugins

https://gerrit.wikimedia.org/r/1277194

gerritbot added a comment.Fri, Apr 24, 9:38 PM

Change #1277194 merged by Bking:

[operations/puppet@production] cloudelastic: add missing extra-analysis plugins

https://gerrit.wikimedia.org/r/1277194

ops-monitoring-bot added a comment.Mon, Apr 27, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.eqiad.wmnet with OS trixie

gerritbot added a comment.Mon, Apr 27, 3:55 PM

Change #1276818 merged by Bking:

[operations/puppet@production] cloudelastic: set role-level hiera for OpenSearch 2/Trixie

https://gerrit.wikimedia.org/r/1276818

bking added a comment.Mon, Apr 27, 4:01 PM

We're making progress! I'm still seeing puppet failures around the installation of the opensearch-madvise package. The package seems to work when installed manually, so I'll take a look at the Puppet code next.

gerritbot added a comment.Mon, Apr 27, 4:33 PM

Change #1277614 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: update allowed plugins list

https://gerrit.wikimedia.org/r/1277614

gerritbot added a comment.Mon, Apr 27, 4:40 PM

Change #1277614 merged by Bking:

[operations/puppet@production] cloudelastic: update allowed plugins list

https://gerrit.wikimedia.org/r/1277614

ops-monitoring-bot added a comment.Mon, Apr 27, 4:55 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.eqiad.wmnet with OS trixie completed:

  • cloudelastic1010 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604271531_bking_2772828_cloudelastic1010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mon, Apr 27, 5:24 PM

Change #1277640 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Another required plugins update patch

https://gerrit.wikimedia.org/r/1277640

gerritbot added a comment.Mon, Apr 27, 5:28 PM

Change #1277640 merged by Bking:

[operations/puppet@production] cloudelastic: Another required plugins update patch

https://gerrit.wikimedia.org/r/1277640

gerritbot added a comment.Mon, Apr 27, 6:41 PM

Change #1277687 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Another ugly plugins patch

https://gerrit.wikimedia.org/r/1277687

gerritbot added a comment.Mon, Apr 27, 6:44 PM

Change #1277687 merged by Bking:

[operations/puppet@production] cloudelastic: Another ugly plugins patch

https://gerrit.wikimedia.org/r/1277687

gerritbot added a comment.Mon, Apr 27, 6:48 PM

Change #1277692 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: Add back the opensearch-ltr plugin

https://gerrit.wikimedia.org/r/1277692

gerritbot added a comment.Mon, Apr 27, 6:50 PM

Change #1277692 merged by Bking:

[operations/puppet@production] cloudelastic: Add back the opensearch-ltr plugin

https://gerrit.wikimedia.org/r/1277692

gerritbot added a comment.Mon, Apr 27, 7:14 PM

Change #1277704 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: remove a plugin path that doesn't exist

https://gerrit.wikimedia.org/r/1277704

gerritbot added a comment.Mon, Apr 27, 7:15 PM

Change #1277704 merged by Bking:

[operations/puppet@production] cloudelastic: remove a plugin path that doesn't exist

https://gerrit.wikimedia.org/r/1277704

gerritbot added a comment.Mon, Apr 27, 7:24 PM

Change #1277708 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: get rid of merge overrides

https://gerrit.wikimedia.org/r/1277708

gerritbot added a comment.Mon, Apr 27, 7:32 PM

Change #1277708 merged by Bking:

[operations/puppet@production] cloudelastic: get rid of merge overrides

https://gerrit.wikimedia.org/r/1277708

ops-monitoring-bot added a comment.Tue, Apr 28, 8:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1009.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Tue, Apr 28, 9:51 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1009.eqiad.wmnet with OS trixie completed:

  • cloudelastic1009 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604282120_bking_3918437_cloudelastic1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Apr 29, 2:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host cloudelastic1008.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Apr 29, 3:16 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host cloudelastic1008.eqiad.wmnet with OS trixie completed:

  • cloudelastic1008 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cloudelastic1008.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl"}
{"cloudelastic1008.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl-public"}
{"cloudelastic1008.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl"}
{"cloudelastic1008.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl-public"}
{"cloudelastic1008.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl"}
{"cloudelastic1008.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl-public"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604290255_ryankemper_4135350_cloudelastic1008.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'name=cloudelastic1008\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-omega\-ssl' set/pooled=no
sudo confctl select 'name=cloudelastic1008\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-omega\-ssl\-public' set/pooled=no
sudo confctl select 'name=cloudelastic1008\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-psi\-ssl' set/pooled=no
sudo confctl select 'name=cloudelastic1008\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-psi\-ssl\-public' set/pooled=no
sudo confctl select 'name=cloudelastic1008\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-chi\-ssl' set/pooled=no
sudo confctl select 'name=cloudelastic1008\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-chi\-ssl\-public' set/pooled=no

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Apr 29, 7:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host cloudelastic1007.eqiad.wmnet with OS trixie

Stashbot added a comment.Wed, Apr 29, 7:39 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-29T07:39:28Z] <ryankemper> T422860 [cloudelastic] Restarted opensearch services on cloudelastic1011 and cloudelastic1012 (needed to pick up missing opensearch plugins, which have already been fixed in puppet) (note: this was done ~2h ago; logged in wrong channel)

ops-monitoring-bot added a comment.Wed, Apr 29, 8:42 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host cloudelastic1007.eqiad.wmnet with OS trixie executed with errors:

  • cloudelastic1007 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cloudelastic1007.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl"}
{"cloudelastic1007.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl-public"}
{"cloudelastic1007.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl"}
{"cloudelastic1007.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl-public"}
{"cloudelastic1007.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl"}
{"cloudelastic1007.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl-public"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'name=cloudelastic1007\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-psi\-ssl' set/pooled=no
sudo confctl select 'name=cloudelastic1007\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-psi\-ssl\-public' set/pooled=no
sudo confctl select 'name=cloudelastic1007\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-chi\-ssl' set/pooled=no
sudo confctl select 'name=cloudelastic1007\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-chi\-ssl\-public' set/pooled=no
sudo confctl select 'name=cloudelastic1007\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-omega\-ssl' set/pooled=no
sudo confctl select 'name=cloudelastic1007\.eqiad\.wmnet,dc=eqiad,cluster=cloudelastic,service=cloudelastic\-omega\-ssl\-public' set/pooled=no

  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudelastic1007.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Wed, Apr 29, 1:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1007.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Apr 29, 2:23 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1007.eqiad.wmnet with OS trixie completed:

  • cloudelastic1007 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604291359_bking_385235_cloudelastic1007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
bking updated the task description. (Show Details)Wed, Apr 29, 2:47 PM
bking closed this task as Resolved.Wed, Apr 29, 7:53 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.

All Cloudelastic hosts are on OpenSearch 2.x/Debian Trixie. We'll focus on the production clusters as time permits; see parent ticket for further updates.

bking updated the task description. (Show Details)Wed, Apr 29, 7:54 PM
bking updated the task description. (Show Details)
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.Fri, May 1, 7:20 AM
MoritzMuehlenhoff created subtask T425422: Some discovery alerts are rejected due to overly long headers.Tue, May 5, 1:31 PM
dcausse moved this task from Incoming to Done on the Discovery-Search (2026.04.06 - 2026.05.01) board.Thu, May 7, 7:23 AM
pfischer moved this task from Done to Reported on the Discovery-Search (2026.04.06 - 2026.05.01) board.Fri, May 8, 10:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits