Page MenuHomePhabricator

Support diversion of testwiki traffic to Ingress at REST Gateway
Open, In Progress, MediumPublic

Description

Background

To support Pretrain, we'll need to divert MediaWiki API traffic at REST Gateway - i.e., testwiki-bound traffic that would normally flow to the upstream mw-api-ext-ro cluster will instead need to flow to mw-pretrain.

Historically, to implement this kind of diversion, we've used dynamic metadata extraction and static LB endpoint subsetting - see, e.g., what we did to support PHP_ENGINE cookie diversion (see https://gerrit.wikimedia.org/r/1194790) or, much earlier, X-Wikimedia-Debug diversion (see https://gerrit.wikimedia.org/r/627588; note that this is not currently used / supported in REST Gateway).

As can be seen in the former, there's a clear maintainability problem with this approach: appropriately handling the explosion of possible metadata value states in the static LB endpoint filter metadata - particularly if we want to simultaneously support diversion for Pretrain, X-Wikimedia-Debug (see T428909#12070270), PHP_ENGINE, and (maybe) some form of transitive RO/RW routing (see T401396#11383272).

Further, note that there's an observability shortcoming: Using LB endpoint subsetting obscures the true upstream, since all potential selections are modeled as endpoints of one "logical" cluster (e.g., everything would still appear to be mw-api-ext-ro).

Is there a better way?

I believe the answer is yes, in the form of the cluster specifier Lua plugin, which would give us a clearer, more easily testable strategy for route-level upstream cluster selection than we could ever achieve with LB endpoint subsetting. Notably, there is plenty of prior art that integrates Lua plugins for various aspects of request processing in REST Gateway - i.e., this is not without precedent, and there are patterns upon which we can build.

If we go this route, then this is also an ideal time to clean up / remove the existing metadata-driven logic as well, and we should aim to coordinate with work in T428625.

Event Timeline

Scott_French changed the task status from Open to In Progress.Aug 1 2026, 1:00 AM
Scott_French triaged this task as Medium priority.
Scott_French moved this task from Inbox to In Progress on the ServiceOps board.

Change #1305236 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] api-gateway: Drop support for debug_hosts

https://gerrit.wikimedia.org/r/1305236

Change #1311874 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] api-gateway: Remove stale test assertion and noop Lua code

https://gerrit.wikimedia.org/r/1311874

Change #1311875 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] api-gateway: Drop support for php_engine_routing

https://gerrit.wikimedia.org/r/1311875

Change #1311963 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] api-gateway: Basic cluster specifier support and Lua plugin

https://gerrit.wikimedia.org/r/1311963

Change #1311964 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] api-gateway: Support x-wikimedia-debug routing

https://gerrit.wikimedia.org/r/1311964

I've attached a couple of proof-of-concept patches that first handle cleanup of the metadata-driven endpoint subset selection and get us to the first non-trivial example of using the plugin (to support X-Wikimedia-Debug). Follow on patches (not yet posted) would reintroduce PHP_ENGINE and (eventually) Pretrain.

It's the 4th patch in the series (https://gerrit.wikimedia.org/r/1311963) that actually introduces the plugin and supporting bits (templates, tests, etc.), while the 5th actually imbues it with real logic.

@Clement_Goubert (and anyone else who might be interested) if I could have your thoughts on the overall approach, that would be greatly appreciated.

Change #1322140 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] api-gateway: Support Host-based diversion in the mw-api plugin

https://gerrit.wikimedia.org/r/1322140

I've rebased the proof-of-concept patch series following the refactoring in T428625, and indeed the conflicts were minimal.

I don't want to get too far ahead of consensus on this implementation path, but given the urgency of making progress on support for Pretrain routing, I've added one additional follow-on patch that provides that.

Rather than being Pretrain-specific, it provides a simple way to generically divert to alternative upstream clusters based on the Host header.

I think consensus is pretty much reached, I don't really see any other way we can do that diversion. I will take a look at the patches in the coming days.

Change #1305236 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Drop support for debug_hosts

https://gerrit.wikimedia.org/r/1305236

Change #1311874 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Remove stale test assertion and noop Lua code

https://gerrit.wikimedia.org/r/1311874

Change #1311875 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Drop support for php_engine_routing

https://gerrit.wikimedia.org/r/1311875

Change #1311963 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Basic cluster specifier support and Lua plugin

https://gerrit.wikimedia.org/r/1311963

I was able to do some basic testing in staging today after merging https://gerrit.wikimedia.org/r/1311963.

That alone is a functional noop - i.e., it does nothing until we actually configure the specifier plugin, and then use it in a route.

I did so temporarily via local edits to values-staging.yaml, using extra_rest_gateway_routes to inject a "sink" route that would shunt all API traffic to cluster_specifier_plugin: mw-api (which currently only returns the default cluster corresponding to the mw-api-ext-ro endpoint).

That seems to have worked as intended. Affected traffic was routed to the expected upstream and envoy logs (which, conveniently, are in debug mode in staging) did not surface any issues.

This bodes well for the remainder of the patch series and for deploying values changes to actually make use of those features.

Change #1311964 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Support x-wikimedia-debug routing

https://gerrit.wikimedia.org/r/1311964

Change #1322140 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Support Host-based diversion in the mw-api plugin

https://gerrit.wikimedia.org/r/1322140

Using a similar strategy to yesterday (T433752#12282637), I was able to test both https://gerrit.wikimedia.org/r/1311964 and https://gerrit.wikimedia.org/r/1322140.

In addition to the same cluster_specifier_plugin: mw-api "sink" route as before, the simplified test setup looked something like:

       tls: true
       port: 4005
+    mwdebug:
+      tls: true
+      port: 4444
+    mw-pretrain:
+      tls: true
+      ingress: true
+      port: 30443
+
+  cluster_specifier_plugins:
+    mw-api:
+      xwd_backends:
+        k8s-mwdebug:
+          discovery_endpoint: mwdebug
+      host_diversion:
+        test.wikipedia.org:
+          discovery_endpoint: mw-pretrain
+      default:
+        discovery_endpoint: mw-api-ext-ro

With that, I was able to verify that:

  1. Requests with x-wikimedia-debug: backend=k8s-mwdebug were diverted to mwdebug as expected (after remembering to open up the network egress policy, that is).
  2. Requests with Host: test.wikipedia.org were diverted to mw-pretrain as expected.
  3. Requests with Host: test.wikipedia.org and x-wikimedia-debugwere diverted to mw-pretrain as expected (i.e., delegating debug routing upstream; in this case, to Ingress).

With that, I think we're in a pretty solid position to start turning this on in production, while monitoring for potential resource impact (?) of putting more Lua on the critical path. I'll start drafting patches for that, starting with XWD. Actually enabling Pretrain diversion will be tracked in the parent tast.

One additional feature that needs added before this task is closed is PHP_ENGINE routing.

Change #1335270 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] api-gateway: Support PHP_ENGINE routing in the mw-api plugin

https://gerrit.wikimedia.org/r/1335270

Change #1335285 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] rest-gateway: Adopt cluster specifier plugin with basic XWD routing

https://gerrit.wikimedia.org/r/1335285