Page MenuHomePhabricator
Create Task
Maniphest T397227

Build and deploy OpenSearch plugins package for updated regex search
Closed, ResolvedPublic
Actions

Description

To support the regex update we need the following two plugin updates live on the production clusters:

  • opensearch-extra 1.3.20-wmf1
  • cirrus-highlighter 1.3.20-wmf1

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update plugins for extended regex supportoperations/software/opensearch/pluginsmaster+44 -34
Customize query in gerrit

Related Objects
Search...

Event Timeline

EBernhardson created this task.Jun 17 2025, 3:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 17 2025, 3:41 PM
EBernhardson added a parent task: T317599: Allow ^ and $ in intitle regex search.Jun 17 2025, 3:41 PM
gerritbot added a comment.Jun 17 2025, 3:42 PM

Change #1143156 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/software/opensearch/plugins@master] Bump ltr plugin to 1.5.4-wmf1-os1.3.20

https://gerrit.wikimedia.org/r/1143156

gerritbot added a project: Patch-For-Review.Jun 17 2025, 3:42 PM
A_smart_kitten subscribed.Jun 17 2025, 9:27 PM
bking subscribed.Jun 18 2025, 3:11 PM
TJones mentioned this in T396529: Reindex All Wikis to enable lots of things.Jun 18 2025, 6:00 PM
Gehel triaged this task as High priority.Jun 20 2025, 8:15 AM
Gehel edited projects, added: Data-Platform-SRE (2025.06.13 - 2025.07.04); removed: Data-Platform-SRE.
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.
gerritbot added a comment.Jul 1 2025, 9:22 PM

Change #1143156 merged by Ryan Kemper:

[operations/software/opensearch/plugins@master] Update plugins for extended regex support

https://gerrit.wikimedia.org/r/1143156

Maintenance_bot removed a project: Patch-For-Review.Jul 1 2025, 9:30 PM
EBernhardson mentioned this in rOSOPeb8e6ad48071: Update plugins for extended regex support.Jul 1 2025, 10:32 PM
bking added a comment.Jul 1 2025, 10:55 PM

The packages are built:

bking@cirrussearch2111:~$ apt policy wmf-opensearch-search-plugins
wmf-opensearch-search-plugins:
  Installed: 1.3.20-4~bullseye
  Candidate: 1.3.20-5~bullseye
  Version table:
     1.3.20-5~bullseye 1001
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/component/opensearch13 amd64 Packages
 *** 1.3.20-4~bullseye 100
        100 /var/lib/dpkg/status

But we still need to install the new packages and roll-restart the cluster to pick up the changes.

Stashbot added a comment.Jul 2 2025, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T13:52:59Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 2 2025, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T13:53:40Z] <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 2 2025, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T14:14:21Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 2 2025, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T14:38:39Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 2 2025, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T14:38:42Z] <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 2 2025, 4:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T16:47:52Z] <inflatador> bking@cumin1002 restarting cirrrussearch codfw T397227

bking added a comment.Jul 2 2025, 10:02 PM

Quick update before the holiday weekend: We've applied the changes and restarted all clusters except for production eqiad. Once we roll-restart EQIAD Monday, the new plugins will be active in all environments.

BTullis edited projects, added: Data-Platform-SRE (2025.07.05 - 2025.07.25); removed: Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM
BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 4 2025, 4:10 PM
Stashbot added a comment.Jul 7 2025, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T14:28:28Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 3:02 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T15:02:29Z] <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 4:00 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T16:00:46Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 4:15 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T16:15:54Z] <bking@cumin1002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 4:16 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T16:16:00Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 5:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T17:12:28Z] <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 5:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T17:58:25Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 5:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T17:58:34Z] <bking@cumin1002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 6:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T18:55:41Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 8:10 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T20:10:05Z] <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 7 2025, 8:11 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T20:11:42Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

bking claimed this task.Jul 7 2025, 9:47 PM
bking moved this task from Backlog - operations to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.
Stashbot added a comment.Jul 7 2025, 10:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-07T22:58:45Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

bking moved this task from In Progress to Needs Review on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 7 2025, 11:06 PM

We've just completed EQIAD, so the plugins should be available in all Search platform OpenSearch clusters. I'm moving this to "Needs Review" so Search Platform SWE's can confirm whether or not the plugins are deployed as expected.

EBernhardson added a comment.EditedJul 8 2025, 3:44 PM

Looks like a couple hosts in codfw still need a restart:

for host in search.svc.{eqiad,codfw}.wmnet cloudelastic.wikimedia.org; do
  for port in 9243 9443 9643; do
    curl https://$host:$port/_cat/plugins
  done
done | grep cirrus-highlighter | grep -v 1.3.20-wmf1 | sort
cirrussearch2062-production-search-codfw cirrus-highlighter                  1.3.20
cirrussearch2062-production-search-psi-codfw cirrus-highlighter                  1.3.20
cirrussearch2069-production-search-codfw cirrus-highlighter                  1.3.20
cirrussearch2069-production-search-psi-codfw cirrus-highlighter                  1.3.20
cirrussearch2075-production-search-codfw cirrus-highlighter                  1.3.20
cirrussearch2075-production-search-psi-codfw cirrus-highlighter                  1.3.20
cirrussearch2076-production-search-codfw cirrus-highlighter                  1.3.20
cirrussearch2076-production-search-psi-codfw cirrus-highlighter                  1.3.20
cirrussearch2089-production-search-codfw cirrus-highlighter                  1.3.20
cirrussearch2089-production-search-psi-codfw cirrus-highlighter                  1.3.20
bking moved this task from Needs Review to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 8 2025, 4:35 PM
Stashbot added a comment.Jul 8 2025, 4:41 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-08T16:40:59Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 8 2025, 5:10 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-08T17:10:33Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

bking closed this task as Resolved.EditedJul 8 2025, 5:15 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.

I've restarted the hosts listed above, and it appears the plugins are now correctly applied:

for host in search.svc.{eqiad,codfw}.wmnet cloudelastic.wikimedia.org; do   for port in 9243 9443 9643; do     curl -s https://$host:$port/_cat/plugins;   done; done | grep cirrus-highlighter | grep -v 1.3.20-wmf1 | sort
(returns nothing)

Since I've confirmed based on the above one-liner, I'll go ahead and close this ticket. Please feel free to reopen if I missed anything.

Stashbot added a comment.Jul 21 2025, 7:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T19:49:25Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 7:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T19:49:29Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 7:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T19:58:57Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 7:59 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T19:59:01Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 8:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T20:01:26Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 9:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T21:25:50Z] <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 9:39 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T21:39:22Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 10:02 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T22:02:08Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 21 2025, 10:22 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-21T22:22:48Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 22 2025, 12:00 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-22T00:00:17Z] <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 23 2025, 7:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-23T19:01:28Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 23 2025, 7:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-23T19:14:08Z] <bking@cumin1002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 23 2025, 7:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-23T19:14:35Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 23 2025, 7:20 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-23T19:20:34Z] <bking@cumin1002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 23 2025, 7:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-23T19:28:44Z] <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 23 2025, 10:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-23T22:53:43Z] <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227

Stashbot added a comment.Jul 23 2025, 11:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-23T23:48:58Z] <ryankemper@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - ryankemper@cumin1002 - T397227

Stashbot added a comment.Jul 24 2025, 1:01 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-24T01:01:51Z] <ryankemper@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - ryankemper@cumin1002 - T397227

BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 28 2025, 5:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits