Page MenuHomePhabricator
Create Task
Maniphest T401396

Revisit backend routing for rest-gateway
Closed, ResolvedPublic
Actions

Description

Currently, backend calls from rest-gateway to MediaWiki APIs are routed to the mw-api-int deployment.
As we're going to shift a significant part of the traffic that is currently served directly by the mw-api-ext deployment, and rest-gateway is a non-transforming service that should never be called internally, it probably makes sense to switch routing MediaWiki APIs to mw-api-ext to maintain a more correct separation of use-cases and align it with naming conventions.

Opening this task for discussion of pros/cons and potential implementation.

Details

Other Assignee
Blake
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
rest-gateway: Point to DC-local mw-api-ext deploymentClément Goubertoperations/deployment-chartsmaster+35 -5
rest-gateway: tweak restbase-compat stats labelHnowlanoperations/deployment-chartsmaster+1 -1
rest-gateway: correct port in networkpolicyHnowlanoperations/deployment-chartsmaster+2 -2
rest-gateway: use mw-api-ext rather than mw-api-int for all APIsHnowlanoperations/deployment-chartsmaster+13 -8
(api|rest)-gateway: always add Via headersHnowlanoperations/deployment-chartsmaster+2 -2
(rest|api)-gateway: change Via header behaviourHnowlanoperations/deployment-chartsmaster+2 -2
(api|rest)-gateway: move Via header definition to responseHnowlanoperations/deployment-chartsmaster+5 -5
trafficserver: multi-dc: use client request host rather than remapHnowlanoperations/puppetproduction+5 -1
(api|rest)-gateway: set Server header if supplied by serviceHnowlanoperations/deployment-chartsmaster+7 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Aug 7 2025, 2:28 PM
CDanis subscribed.Aug 7 2025, 2:29 PM
gerritbot added a comment.Sep 16 2025, 11:59 AM

Change #1188758 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] (api|rest)-gateway: set Server header if supplied by service

https://gerrit.wikimedia.org/r/1188758

gerritbot added a project: Patch-For-Review.Sep 16 2025, 11:59 AM
gerritbot added a comment.Sep 17 2025, 8:48 AM

Change #1188758 merged by jenkins-bot:

[operations/deployment-charts@master] (api|rest)-gateway: set Server header if supplied by service

https://gerrit.wikimedia.org/r/1188758

gerritbot added a comment.Sep 17 2025, 9:24 AM

Change #1189121 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] (api|rest)-gateway: move Via header definition to response

https://gerrit.wikimedia.org/r/1189121

gerritbot added a comment.Sep 17 2025, 11:13 AM

Change #1189132 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: multi-dc: use client request host rather than remap

https://gerrit.wikimedia.org/r/1189132

gerritbot added a comment.Sep 23 2025, 10:25 AM

Change #1189132 merged by Clément Goubert:

[operations/puppet@production] trafficserver: multi-dc: use client request host rather than remap

https://gerrit.wikimedia.org/r/1189132

gerritbot added a comment.Sep 23 2025, 10:36 AM

Change #1189121 merged by jenkins-bot:

[operations/deployment-charts@master] (api|rest)-gateway: move Via header definition to response

https://gerrit.wikimedia.org/r/1189121

gerritbot added a comment.Sep 23 2025, 10:54 AM

Change #1190628 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] (rest|api)-gateway: change Via header behaviour

https://gerrit.wikimedia.org/r/1190628

gerritbot added a comment.Sep 23 2025, 10:58 AM

Change #1190628 merged by jenkins-bot:

[operations/deployment-charts@master] (rest|api)-gateway: change Via header behaviour

https://gerrit.wikimedia.org/r/1190628

Maintenance_bot removed a project: Patch-For-Review.Sep 23 2025, 11:31 AM
gerritbot added a comment.Sep 23 2025, 12:08 PM

Change #1190653 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] (api|rest)-gateway: always add Via headers

https://gerrit.wikimedia.org/r/1190653

gerritbot added a project: Patch-For-Review.Sep 23 2025, 12:08 PM
gerritbot added a comment.Sep 23 2025, 12:15 PM

Change #1190653 merged by jenkins-bot:

[operations/deployment-charts@master] (api|rest)-gateway: always add Via headers

https://gerrit.wikimedia.org/r/1190653

Maintenance_bot removed a project: Patch-For-Review.Sep 23 2025, 12:31 PM
HCoplin-WMF moved this task from Now to Supporting on the MediaWiki-API-Platform-Team (MWI-Roadmap) board.Sep 23 2025, 10:28 PM
gerritbot added a comment.Oct 3 2025, 10:28 AM

Change #1193389 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: use mw-api-ext rather than mw-api-int for all APIs

https://gerrit.wikimedia.org/r/1193389

gerritbot added a project: Patch-For-Review.Oct 3 2025, 10:28 AM
hnowlan changed the task status from Open to In Progress.Oct 3 2025, 10:54 AM
hnowlan claimed this task.
hnowlan mentioned this in T406318: rest.php via rest-gateway production rollout.Oct 3 2025, 11:23 AM
gerritbot added a comment.Oct 7 2025, 11:01 AM

Change #1193389 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: use mw-api-ext rather than mw-api-int for all APIs

https://gerrit.wikimedia.org/r/1193389

gerritbot added a comment.Oct 7 2025, 11:08 AM

Change #1194172 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: correct port in networkpolicy

https://gerrit.wikimedia.org/r/1194172

gerritbot added a comment.Oct 7 2025, 11:16 AM

Change #1194172 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: correct port in networkpolicy

https://gerrit.wikimedia.org/r/1194172

gerritbot added a comment.Oct 7 2025, 11:29 AM

Change #1194176 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: tweak restbase-compat stats label

https://gerrit.wikimedia.org/r/1194176

gerritbot added a comment.Oct 7 2025, 11:41 AM

Change #1194176 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: tweak restbase-compat stats label

https://gerrit.wikimedia.org/r/1194176

hnowlan closed this task as Resolved.Oct 7 2025, 11:55 AM

We're now using mw-api-ext as appropriate for rest.php and mw-api-related APIs.

Maintenance_bot removed a project: Patch-For-Review.Oct 7 2025, 12:30 PM
Clement_Goubert reopened this task as In Progress.Nov 18 2025, 11:45 AM
Clement_Goubert triaged this task as Medium priority.
Clement_Goubert added subscribers: Scott_French, akosiaris.

Reopening for followup discussion of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1204865

Once this patch is merged, rest-gateway will use mw-api-ext-ro as a stand-in for dc-local queries. In the normal operation state, this means that primary-bound requests as determined by ATS will be directed to the primary DC instance of the rest-gateway, hitting the primary DC instance of mw-api-ext.
However, this poses an interesting problem during non-standard operation, for instance during the first day of the switchover when services have been migrated to the future-primary but not MediaWiki, or if we depool the primary DC from mw-api-ext-ro. In these cases, the cross-datacenter round-trip goes down a few levels and happens at database request time. This obviously multiplies the latency penalty, as a single HTTP request to the backend will almost certainly do multiple DB requests.

One of the potential solutions suggested in the linked CR from @Scott_French :

What if multi-dc.lua added a header to the request indicating whether it should be routed to the primary? (i.e., in addition to using that decision for its own remapping)

An upstream system that functions as a proxy could then (optionally) use that information to select an upstream.

In rest-gateway, we'd use a similar trick to what I was doing with PHP_ENGINE, using envoy.filters.http.header_to_metadata to drive LB subset selection.

Even when we're in the weird ~ 24h where MediaWiki's primary DC differs from that of rest-gateway.discovery.wmnet, the request would still go to the right place (i.e., follow the subset selection to mw-api-ext.discovery.wmnet).

In terms of making routing easier to reason about, I think it's a net win if "primary-bound" can follow the request like this, rather than only applying at the first hop. However, I need to give it some more thought 😊

This is in line with my own reflections, as we will increasingly have to access that information for routing decisions as we move forwards with future T384944: Implement the MediaWiki Singleversion experimentation outlined in FY 2024-2025 WE6.2.5 and WE6.2.9 Hypothesis related developments (cc @akosiaris)

gerritbot added a comment.Nov 18 2025, 11:46 AM

Change #1204865 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Point to DC-local mw-api-ext deployment

https://gerrit.wikimedia.org/r/1204865

gerritbot added a project: Patch-For-Review.Nov 18 2025, 11:46 AM
gerritbot added a comment.Nov 20 2025, 10:39 AM

Change #1204865 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Point to DC-local mw-api-ext deployment

https://gerrit.wikimedia.org/r/1204865

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2025, 11:31 AM
Clement_Goubert moved this task from Incoming 🐫 to 🥋Good First Task on the serviceops-deprecated board.Nov 20 2025, 4:46 PM
Clement_Goubert added a subscriber: Blake.

I'd like to do the actual implementation of this with @Blake in the upcoming months.

MLechvien-WMF edited projects, added: ServiceOps, ServiceOps-good-first-task, ServiceOps-SharedInfra; removed: serviceops-deprecated.Jan 16 2026, 4:35 PM
MLechvien-WMF subscribed.

@Clement_Goubert I suspect things have changed, could you confirm that it's still valid?

If still valid it should probably be moved to Scheduled work

MLechvien-WMF reassigned this task from hnowlan to Clement_Goubert.Jan 16 2026, 4:35 PM
MLechvien-WMF moved this task from Inbox to Needs Info / Blocked on the ServiceOps board.
Clement_Goubert moved this task from Needs Info / Blocked to Scheduled (this Q) on the ServiceOps board.Jan 17 2026, 10:51 AM
Clement_Goubert updated Other Assignee, added: Blake.

Yes, we should try and get this done before the switchover.

Clement_Goubert closed this task as Resolved.Jun 9 2026, 2:17 PM
Scott_French mentioned this in T433752: Support diversion of testwiki traffic to Ingress at REST Gateway.Aug 1 2026, 1:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits