Page MenuHomePhabricator
Create Task
Maniphest T417698

Ensure select dse-k8s-hosted microservices can run active/active
Closed, ResolvedPublic
Actions

Description

While we added a second dse-k8s cluster in CODFW, we never completed the necessary steps for our hosted services to run active/active.

There's not much use to having a second cluster if we can't do active/active for services, so I'm creating this ticket to:

  • Complete all steps necessary for select dse-k8s services to run active/active*
  • Verify operation

*Not all services can or should run active/active. For the purposes of this ticket, we're scoping to opensearch-ipoid and opensearch-test.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add new active/active discovery records for dse-k8s opensearch prod nsoperations/dnsmaster+3 -3
Add new active/active discovery records for dse-k8s opensearch test nsoperations/dnsmaster+3 -3
Add new active-active discovery records for dse-k8soperations/dnsmaster+3 -0
Add new active-active discovery service for dse-k8soperations/puppetproduction+5 -1
Customize query in gerrit

Related Objects

Event Timeline

bking created this task.Feb 17 2026, 8:13 PM
bking triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 17 2026, 8:13 PM
bking renamed this task from Deploy active/active for select dse-k8s microservices to Ensure select dse-k8s-hosted microservices can run active/active.Feb 17 2026, 8:19 PM
bking edited projects, added Data-Platform-SRE (2026-02-13 - 2026-03-06); removed Data-Platform-SRE.
bking updated the task description. (Show Details)
bking mentioned this in T416345: Investigate higher-than-expected latency on opensearch-ipoid API responses.Feb 17 2026, 8:43 PM
bking updated the task description. (Show Details)Feb 17 2026, 8:55 PM
bking claimed this task.Mar 5 2026, 6:44 PM
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
bking added subscribers: ssingh, brouberol, RKemper.
bking added a comment.EditedMar 5 2026, 7:26 PM

This morning, we attempted to enable active-active via this maintenance plan and it had to be rolled back.

After some discussion with Traffic, we do not think the previous approach is going to work. One of the previous approaches we discussed is creating a new domain and service, distinct from disc-k8s-ingress-dse, that we can use for active-active services.

I'll begin the discussion with my team/Traffic in Slack and update the ticket as needed.

gerritbot added a comment.Mar 5 2026, 8:52 PM

Change #1248605 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] WIP: Add new active-active discovery service for dse-k8s

https://gerrit.wikimedia.org/r/1248605

gerritbot added a project: Patch-For-Review.Mar 5 2026, 8:52 PM
gerritbot added a comment.Mar 5 2026, 9:54 PM

Change #1248625 had a related patch set uploaded (by Bking; author: Bking):

[operations/dns@master] WIP: Add new active-active discovery records for dse-k8s

https://gerrit.wikimedia.org/r/1248625

Gehel edited projects, added Data-Platform-SRE (2026-03-06 - 2026-03-27); removed Data-Platform-SRE (2026-02-13 - 2026-03-06).Mar 6 2026, 2:02 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
gerritbot added a comment.Mar 10 2026, 7:49 PM

Change #1250063 had a related patch set uploaded (by Bking; author: Bking):

[operations/dns@master] Add new active/active discovery records for dse-k8s opensearch test ns

https://gerrit.wikimedia.org/r/1250063

gerritbot added a comment.Mar 10 2026, 8:10 PM

Change #1250068 had a related patch set uploaded (by Bking; author: Bking):

[operations/dns@master] Add new active/active discovery records for dse-k8s opensearch prod ns

https://gerrit.wikimedia.org/r/1250068

gerritbot added a comment.Mon, Mar 23, 2:08 PM

Change #1248605 merged by Bking:

[operations/puppet@production] Add new active-active discovery service for dse-k8s

https://gerrit.wikimedia.org/r/1248605

gerritbot added a comment.Mon, Mar 23, 2:32 PM

Change #1248625 merged by Ssingh:

[operations/dns@master] Add new active-active discovery records for dse-k8s

https://gerrit.wikimedia.org/r/1248625

gerritbot added a comment.Mon, Mar 23, 2:48 PM

Change #1250063 merged by Bking:

[operations/dns@master] Add new active/active discovery records for dse-k8s opensearch test ns

https://gerrit.wikimedia.org/r/1250063

gerritbot added a comment.Mon, Mar 23, 2:55 PM

Change #1250068 merged by Bking:

[operations/dns@master] Add new active/active discovery records for dse-k8s opensearch prod ns

https://gerrit.wikimedia.org/r/1250068

bking updated the task description. (Show Details)Mon, Mar 23, 3:29 PM
Maintenance_bot removed a project: Patch-For-Review.Mon, Mar 23, 3:30 PM
bking closed this task as Resolved.EditedMon, Mar 23, 3:39 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.

After merging the above changes, I can confirm that opensearch-ipoid.discovery.wmnet is now correctly balanced between EQIAD and CODFW, instead of just going to EQIAD.

This change has improved iPoid's P50 response times as well. Closing...

bking mentioned this in T420638: Investigate elevated P99/connection errors for opensearch-ipoid beginning 17 March 2026 at ~1600 UTC.Mon, Mar 23, 3:43 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.Fri, Mar 27, 8:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits