Page MenuHomePhabricator
Create Task
Maniphest T422378

Move OpenSearch-iPoid behind TLS/Update to OpenSearch 3.x
Open, In Progress, MediumPublic
Actions

Description

OpenSearch iPoid was our first tenant on the OpenSearch-on-K8s platform.

Since its launch, we have added a couple of features:

  • TLS termination at the Envoy level : this allows our OpenSearch clusters to integrate with the service mesh, reducing connection startup times for clients and enabling connection metrics that will be useful for troubleshooting and SLO creation.
  • OpenSearch 3.x : The latest major version of OpenSearch, offers performance updates and newer features. Another one of our clusters, opensearch-semantic-search , has been running this version for weeks and we haven't seen any issues.

Creating this ticket to:

  • Apply envoy-level TLS termination to opensearch-ipoid - tentatively scheduled for completed 9 Apr 2026 @1330 UTC
  • Update opensearch-ipoid to OpenSearch 3.x - schedule subject to approval by @kostajh

/PSI

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
opensearch-ipoid: remove version pinBkingoperations/deployment-chartsmaster+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Apr 6 2026, 1:54 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 6 2026, 1:54 PM
bking mentioned this in T418175: Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform.Apr 6 2026, 1:55 PM
bking updated the task description. (Show Details)Apr 6 2026, 1:57 PM
gerritbot added a comment.Apr 6 2026, 2:00 PM

Change #1268222 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] opensearch-ipoid: remove version pin

https://gerrit.wikimedia.org/r/1268222

gerritbot added a project: Patch-For-Review.Apr 6 2026, 2:00 PM
bking updated the task description. (Show Details)Apr 8 2026, 2:41 PM
bking updated the task description. (Show Details)Apr 8 2026, 2:43 PM
gerritbot added a comment.Apr 9 2026, 1:19 PM

Change #1268222 merged by Bking:

[operations/deployment-charts@master] opensearch-ipoid: remove version pin

https://gerrit.wikimedia.org/r/1268222

bking updated the task description. (Show Details)Apr 9 2026, 9:48 PM
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:17 AM
bking updated the task description. (Show Details)May 12 2026, 5:18 PM
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
bking updated the task description. (Show Details)
bking added a comment.May 12 2026, 5:27 PM

Tentative maintenance plan:

  • Merge changes to deployment-charts that include the OpenSearch 3.x image
  • Failover opensearch-ipoid to EQIAD only using DNS discovery (DPE SRE)
  • Upgrade to OpenSearch 3.x in CODFW (DPE SRE)
  • Confirm application is working on OpenSearch 3.x (PSI)
  • Repeat in EQIAD

Rollback plan:

  • Remove failed cluster in CODFW (assuming we have a failure there first) (DPE SRE)
  • Rollback patch that updates Docker image (DPE SRE)
  • Regenerate indices using Airflow DAG (PSI)
  • Confirm newly-provisioned cluster has correct data (PSI)
  • Re-enable CODFW in DNS discovery (DPE SRE)
bking changed the task status from Open to In Progress.May 14 2026, 7:06 PM
bking triaged this task as Medium priority.
Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE (2026-04-24 - 2026-05-15).Jun 5 2026, 1:27 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Gehel edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE (2026-06-05 - 2026-06-26).Jul 3 2026, 9:47 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits