From running https://gist.github.com/AaronSchulz/80e2b0f97ae8ff6caf52cb425f8e431c , I'm getting one new unexpected header on every test request:

WARNING: GET https://test2.wikipedia.org/w/rest.php/v1/search/page?q=Aleph&limit=1 from anon
REASON: mwapi vs gateway header differences
!+ content-security-policy: default-src 'none'; frame-ancestors 'none'
~+ accept-ranges: bytes
~- server: mw-api-ext.codfw.main-6647cc6f69-tzfhj
~+ server: mw-api-int.codfw.main-7c66456f9f-9rvwm

...

WARNING: POST https://test2.wikipedia.org/w/rest.php/v1/transform/wikitext/to/lint/Aleph/598651 from user
REASON: mwapi vs gateway header differences
~- date: Wed, 24 Sep 2025 16:20:04 GMT
~+ date: Wed, 24 Sep 2025 16:20:03 GMT
!+ content-security-policy: default-src 'none'; frame-ancestors 'none'
~- server: mw-api-ext.codfw.main-6647cc6f69-275zt
~+ server: mw-api-int.codfw.main-7c66456f9f-bzc4f

I'm also seeing extra headers for all the transform POST endpoints that give HTML, e.g.:

WARNING: POST https://test2.wikipedia.org/w/rest.php/v1/transform/wikitext/to/html/Aleph from anon
REASON: mwapi vs gateway header differences
!+ x-frame-options: SAMEORIGIN
!+ x-xss-protection: 1; mode=block
~- server: mw-api-ext.codfw.main-6647cc6f69-59k4s
~+ server: mw-api-int.codfw.main-7c66456f9f-kg5cm
!+ referrer-policy: origin-when-cross-origin
!+ content-security-policy: default-src 'none'; frame-ancestors 'none'
!+ access-control-allow-methods: GET,HEAD
~- date: Wed, 24 Sep 2025 17:19:14 GMT
~+ date: Wed, 24 Sep 2025 17:19:13 GMT
!+ access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
!+ access-control-expose-headers: etag

In T405368#11211211, @aaron wrote:

I'm also seeing extra headers for all the transform POST endpoints that give HTML, e.g.:

WARNING: POST https://test2.wikipedia.org/w/rest.php/v1/transform/wikitext/to/html/Aleph from anon
REASON: mwapi vs gateway header differences
!+ x-frame-options: SAMEORIGIN
!+ x-xss-protection: 1; mode=block
~- server: mw-api-ext.codfw.main-6647cc6f69-59k4s
~+ server: mw-api-int.codfw.main-7c66456f9f-kg5cm
!+ referrer-policy: origin-when-cross-origin
!+ content-security-policy: default-src 'none'; frame-ancestors 'none'
!+ access-control-allow-methods: GET,HEAD
~- date: Wed, 24 Sep 2025 17:19:14 GMT
~+ date: Wed, 24 Sep 2025 17:19:13 GMT
!+ access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
!+ access-control-expose-headers: etag

Looking at "- name: mw-api-int" in rest-gateway/values.yaml, all the '(.+)/v1/<stuff>' stuff is really looking for "/<domain>/v1/<stuff>', right? It's just using (.+) for convenience, then.
A path like "/w/rest.php/v1/transform/wikitext/to/html" matches both:

• '/w/rest.php/v1/transform(.*)', from the "transform " route of "mw-rest-php", and
• '(.+)/v1/transform/wikitext/to/html(.*)', from the "'(.+)/v1/transform/wikitext/to/html(.*)'" route of "mw-api-int"

The second one is generated first in the envoy routes, so it "wins", hence /w/rest.php/v1/transform/wikitext/to/html requests from clients get the mw-api-int treatment meant for /api/rest_v1/transform/wikitext/to/html requests from clients.

That explains why rb_response_headers behavior applies and we get these headers. The "transform" route isn't getting reached here.

Change #1191333 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Tighten non mw-rest-php matches

https://gerrit.wikimedia.org/r/1191333

Change #1191344 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] (api|rest)-gateway: Add option to disable CSP, disable for rest.php

https://gerrit.wikimedia.org/r/1191344

Change #1191333 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Tighten non mw-rest-php matches

https://gerrit.wikimedia.org/r/1191333

Change #1191444 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Relax leading slash match

https://gerrit.wikimedia.org/r/1191444

Change #1191444 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Tighten non mw-rest-php matches

https://gerrit.wikimedia.org/r/1191444

In T405368#11211609, @aaron wrote:

In T405368#11211211, @aaron wrote:

I'm also seeing extra headers for all the transform POST endpoints that give HTML, e.g.:

WARNING: POST https://test2.wikipedia.org/w/rest.php/v1/transform/wikitext/to/html/Aleph from anon
REASON: mwapi vs gateway header differences
!+ x-frame-options: SAMEORIGIN
!+ x-xss-protection: 1; mode=block
~- server: mw-api-ext.codfw.main-6647cc6f69-59k4s
~+ server: mw-api-int.codfw.main-7c66456f9f-kg5cm
!+ referrer-policy: origin-when-cross-origin
!+ content-security-policy: default-src 'none'; frame-ancestors 'none'
!+ access-control-allow-methods: GET,HEAD
~- date: Wed, 24 Sep 2025 17:19:14 GMT
~+ date: Wed, 24 Sep 2025 17:19:13 GMT
!+ access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
!+ access-control-expose-headers: etag

Looking at "- name: mw-api-int" in rest-gateway/values.yaml, all the '(.+)/v1/<stuff>' stuff is really looking for "/<domain>/v1/<stuff>', right? It's just using (.+) for convenience, then.
A path like "/w/rest.php/v1/transform/wikitext/to/html" matches both:

• '/w/rest.php/v1/transform(.*)', from the "transform " route of "mw-rest-php", and
• '(.+)/v1/transform/wikitext/to/html(.*)', from the "'(.+)/v1/transform/wikitext/to/html(.*)'" route of "mw-api-int"

The second one is generated first in the envoy routes, so it "wins", hence /w/rest.php/v1/transform/wikitext/to/html requests from clients get the mw-api-int treatment meant for /api/rest_v1/transform/wikitext/to/html requests from clients.

That explains why rb_response_headers behavior applies and we get these headers. The "transform" route isn't getting reached here.

Sorry it took that long, but that is now fixed. When the routes for rest.php are stabilized, we should compile your list of tests + create one for the other routes handled by the rest-gateway so we can

1. document the configuration for what should match
2. enrich the fork of your test script I started to test these changes (no header-diff check, just response header validation against a matches for ex-restbase routes, normal microservices routes like mobileapps and rest.php routes)

I'll add some more test for the endpoints (to by gist) I didn't cover due to permissions or a lack of data. Even just expecting the error would still be useful for comparing headers.

In T405368#11225897, @aaron wrote:

I'll add some more test for the endpoints (to by gist) I didn't cover due to permissions or a lack of data. Even just expecting the error would still be useful for comparing headers.

I updated the gist with a few such routes (e.g. ipinfo/checkuser/oauth). Nothing new found.

In T405368#11224527, @Clement_Goubert wrote:

Sorry it took that long, but that is now fixed. When the routes for rest.php are stabilized, we should compile your list of tests + create one for the other routes handled by the rest-gateway so we can

1. document the configuration for what should match
2. enrich the fork of your test script I started to test these changes (no header-diff check, just response header validation against a matches for ex-restbase routes, normal microservices routes like mobileapps and rest.php routes)

To be clear, were you thinking of doing that as post-migration cleanup or before the migration?

In T405368#11226157, @aaron wrote:

In T405368#11224527, @Clement_Goubert wrote:

Sorry it took that long, but that is now fixed. When the routes for rest.php are stabilized, we should compile your list of tests + create one for the other routes handled by the rest-gateway so we can

1. document the configuration for what should match
2. enrich the fork of your test script I started to test these changes (no header-diff check, just response header validation against a matches for ex-restbase routes, normal microservices routes like mobileapps and rest.php routes)

To be clear, were you thinking of doing that as post-migration cleanup or before the migration?

I was thinking of doing it once we're pretty confident the routes for rest.php are not going to change much, so we don't spend too much time on it, but I can be persuaded any which way.

Change #1191344 merged by jenkins-bot:

[operations/deployment-charts@master] (api|rest)-gateway: Add option to disable CSP, disable for rest.php

https://gerrit.wikimedia.org/r/1191344

In T405368#11211211, @aaron wrote:

I'm also seeing extra headers for all the transform POST endpoints that give HTML, e.g.:

WARNING: POST https://test2.wikipedia.org/w/rest.php/v1/transform/wikitext/to/html/Aleph from anon
REASON: mwapi vs gateway header differences
!+ x-frame-options: SAMEORIGIN
!+ x-xss-protection: 1; mode=block
~- server: mw-api-ext.codfw.main-6647cc6f69-59k4s
~+ server: mw-api-int.codfw.main-7c66456f9f-kg5cm
!+ referrer-policy: origin-when-cross-origin
!+ content-security-policy: default-src 'none'; frame-ancestors 'none'
!+ access-control-allow-methods: GET,HEAD
~- date: Wed, 24 Sep 2025 17:19:14 GMT
~+ date: Wed, 24 Sep 2025 17:19:13 GMT
!+ access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
!+ access-control-expose-headers: etag

content-security-policy has been toggled off for the rest.php response headers. Between this and the correction of the overlapping routes (great catch again!) are there any outstanding discrepancies?

@aaron can you please rerun the tests today to see if there is anything new or outstanding? If we are looking good, we should plan to dial up on Monday:

• Group 0 = internal + mw.org (100% or large %; maybe exclude mw.org or ramp more slowly)
• Group 1 = smaller wikis (maybe a negligible %)

Once those tests are done/everything looks good, we should be able to ramp up quickly on everything else.

@aaron -- just confirming if you were able to rerun the tests? Could you please let us know if you found anything new, or if we're good to go?