I think the simplest way to go about this is to remove X-Wikimedia-Debug from the plugin chain for api.wikimedia.org/{core,service,feed} in the ATS configuration.
It is not currently needed as even the /core API endpoints can't currently be tested on these paths using X-W-D.

Change #1306643 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] trafficserver::backend: Remove X-W-D for api.w.o

https://gerrit.wikimedia.org/r/1306643

I think given the API portal is deprecated and the /core/ APIs are in maintenance mode I would be of the opinion that we can just drop api-gateway-specific support for XWD entirely. It was mostly used for test deployments of the portal (T262396) - if the linkrecommendation folks are using it then that's another matter, but if the bug in T430600 was an accidental discovery I would vote for dropping the support entirely.

Change #1306643 merged by Clément Goubert:

[operations/puppet@production] trafficserver::backend: Remove X-W-D for api.w.o

https://gerrit.wikimedia.org/r/1306643

In T428909#12069858, @hnowlan wrote:

I think given the API portal is deprecated and the /core/ APIs are in maintenance mode I would be of the opinion that we can just drop api-gateway-specific support for XWD entirely. It was mostly used for test deployments of the portal (T262396) - if the linkrecommendation folks are using it then that's another matter, but if the bug in T430600 was an accidental discovery I would vote for dropping the support entirely.

I think we do need to support it eventually given the REST Gateway will be used to implement part of the new canonical base URL for APIs, and they may need to be tested on mw-debug while still going through the REST Gateway.

Here's my take.

X-Wikimedia-Debug handling for REST endpoints

Commmon to all solutions

• Remove X-W-D based diversion in ATS for anything going to the REST Gateway.
• Pass the header on to the REST Gateway untouched.

REST Gateway based

• Add mw-debug deployments as backends in the REST Gateway.
• Configure the REST Gateway's envoy to switch backends based on that header value using router.cluster_specifiers.lua.

Pros:

• Can be done right now.
• Centralizes header based decisions for APIs in the REST Gateway.
• Allows to extend the behaviour to allow using staging deployments of non-MediaWiki services as backends when X-W-D is set.
• Allows differential ratelimiting for mw-debug hits if we want to

Cons:

• Adds complexity to the REST Gateway configuration (service-level complexity).
• Doesn't enable X-W-D header for internal use in MediaWiki backend API calling.

Mesh based (current)

• Have the REST Gateway pass the X-W-D header through untouched.
• Configure the current mesh terminator to switch backends based on the header value using once again router.cluster_specifiers.lua.

Pros:

• Moves X-W-D handling as far down the stack as possible.
• Enables use of X-W-D by non-MediaWiki services through the mesh module.

Cons:

• Adds complexity to the mesh's envoy configuration, which is used by all services (framework-level complexity).
• Probably doesn't fit within a future "single-version routing" environment.
• Doesn't allow differential ratelimiting for mw-debug without a specific implementation in the REST Gateway

Ingress based (future)

• Same as Mesh based, but configuration is in ingress instead of mesh terminator.

Pros:

• Moves X-W-D handling as far down the stack as possible.
• Works within a "single-version routing" environment.

Cons:

• Can't be done now as MediaWiki deployments are generally not behind ingress.
• Doesn't work for services not behind ingress.
• Doesn't allow differential ratelimiting for mw-debug without a specific implementation in the REST Gateway
• Ingress configuration duplication will be needed for non-MediaWiki services as it is currently not modularized as the mesh configuration is.

For now, I've removed any X-W-D handling on api.wikimedia.org. Other endpoints status:

• /w/rest.php: There is an argument to remove it from this endpoint because the REST Gateway rewrites the paths. This means hitting mw-debug directly will probably break.
• /api/rest_v1: This endpoint does not have X-W-D handling configured.
• /w/api.php (Action API): This endpoint has X-W-D handling configured, but the REST Gateway does not meaningfully rewrite the paths. Hitting mw-debug directly works identically to going through the REST Gateway, but bypasses rate limiting.

I think until we actually implement correct X-W-D handling for REST endpoints, we should remove it from /w/rest.php (because it is broken). Being able to hit the Action API in mw-debug directly seems important enough to keep.

Thanks for working on this, @Clement_Goubert!

Change #1306905 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] trafficserver::backend: Remove X-W-D for /w/rest.php

https://gerrit.wikimedia.org/r/1306905

Nice analysis in T428909#12070270 @Clement_Goubert.

+1 to your assessment in T428909#12070272. If X-W-D diversion at ATS breaks /w/rest.php calls due to the missing rewrites, then the least-surprising option is to disable it as you've proposed.

As for introducing X-W-D support for rest-gateway routed traffic, I would advocate for what you described as the REST Gateway based option, as a fairly ideal near-term solution. That would align with one of the options I had in mind for supporting Pretrain routing (T427666#12052297), and (IMO) nicely encapsulates complexity around diversion to the "last hop" toward MediaWiki (vs., e.g., pushing it out into the service mesh).

Also, +1 to wider Ingress adoption being where we end up eventually, most likely in the form of Single-Version routing. That's subsidiary goal of how we're configuring Pretrain routing, as a next incremental step toward what Single-Version-on-Ingress would look like (e.g., VirtualService-based X-W-D diversion).

Copying my comment from gerrit here for ease of retrieval:

In 1306905, @Clement_Goubert wrote:

This affects a majority of the old RESTBase endpoints, namely:

page-metadata-revision
page-metadata-title
page-html-revision
page-html-title
page-lint-revision
page-lint-title
lists
all rest specs files

Unaffected from that group are the transform endpoints:

transform_wikitext_to_html
transform_html_to_wikitext
transform_wikitext_to_lint

These are in the mw-api-ext section of the REST Gateway configuration.

This would also affect any endpoint going to a non-mediawiki service that has rewrites in the REST Gateway, like proton, wikifeeds, or most *-analytics endpoints.

The endpoints in mw-rest-auth and mw-rest-php are all fine except for the sandbox redirects.

I suspect API endpoints are not tested often using X-W-D through the REST Gateway given they're not yet used by on-wiki client-side JS much. As far as I can tell from git blame we introduced that bug in September 2024 at the latest, probably before that.

This is definitely a temporary stopgap measure to make sure behavior is consistent for a general API type (i.e. no /w/rest.php or api.w.o endpoints support X-W-D, /w/api.php do).

In the future I think we need to support X-W-D for anything going through the REST Gateway by removing that logic from ATS for all API endpoints and allowing backend switching in the REST Gateway to whatever the service owners want:

• Do nothing and just hit the normal prod endpoint
• Pass the X-W-D header through to the normal prod endpoint and let the service do its thing
• Send the request to mw-debug or to a staging deployment for non-mediawiki services
• etc.

I will post a message to MediaWiki-Engineering and update the X-W-D wikitech page for clarity, but I think we should decide on a way forward to globally fix X-W-D handling sooner rather than later, especially with the unified developer portal and associated client-side sandboxes coming soon.

Disregard the list of old RESTBase endpoints, actually, those are under /api/rest_v1/. The point stands that it's confusing as heck :D

Change #1306905 merged by Clément Goubert:

[operations/puppet@production] trafficserver::backend: Remove X-W-D for /w/rest.php

https://gerrit.wikimedia.org/r/1306905

Cross-posting from T431838: mw-parsoid endpoints are returning 404 for the rest.php endpoints used by rt-testing: https://gerrit.wikimedia.org/r/1306905 seems to have broken the new Parsoid RT testing setup, which now relies on X-Wikimedia-Debug in order to access Parsoid REST API paths hosted on mw-parsoid from Cloud VPS. Summary in T431838#12109862.

Change #1309689 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] trafficserver::backend: Restore X-W-D for /w/rest.php

https://gerrit.wikimedia.org/r/1309689

Mentioned in SAL (#wikimedia-operations) [2026-07-13T14:05:17Z] <swfrench-wmf> disable-puppet on A:cp for ATS config change - T428909 T431838

Change #1309689 merged by Scott French:

[operations/puppet@production] trafficserver::backend: Restore X-W-D for /w/rest.php

https://gerrit.wikimedia.org/r/1309689

Mentioned in SAL (#wikimedia-operations) [2026-07-13T14:14:37Z] <swfrench-wmf> start rolling run-puppet-agent on A:cp for ATS config change - T428909 T431838

The revert of https://gerrit.wikimedia.org/r/1306905 finished rolling out at ~ 14:45 UTC. I've added a note to https://wikitech.wikimedia.org/wiki/WikimediaDebug#Limitations about the fact that X-W-D has been temporarily reenabled for REST API paths, and what the implications are (i.e., paths relying on gateway rewrites will not work).

Change #1310103 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] trafficserver: Remove XWD routing for /w/rest.php mw-debug backend

https://gerrit.wikimedia.org/r/1310103

Change #1310103 merged by Effie Mouzeli:

[operations/puppet@production] trafficserver: Remove XWD routing for /w/rest.php mw-debug backend

https://gerrit.wikimedia.org/r/1310103

Tested https://gerrit.wikimedia.org/r/1310103

• Got a 200 OK for

curl -v -X GET 'https://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/wikitext/Dog' -H "X-Wikimedia-Debug: backend=k8s-mw-parsoid-codfw"

• Got a 404 for

curl -v -X GET 'https://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/wikitext/Dog' -H "X-Wikimedia-Debug: backend=k8s-mw-mwdebug"

cc @Jgiannelos

Looks ok:

curl -v -o /dev/null 'https://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/Dog' -H 'X-Wikimedia-Debug: backend=k8s-mw-parsoid-eqiad' 2>&1 | grep "< server"
< server: mw-parsoid.eqiad.pinkllama-6587bc84c6-vjq8v
< server-timing: WMF-Uniq;desc="we-1-8-tempuser-post-edit=treatment;";co_id;desc="282594226"

curl -v -o /dev/null 'https://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/Dog' -H 'X-Wikimedia-Debug: backend=k8s-mw-parsoid-codfw' 2>&1 | grep "< server"
< server: mw-parsoid.codfw.pinkllama-99bb97887-z6t46
< server-timing: cache;desc="pass", host;desc="cp3070";co_id;desc="1789763765"