@ssingh @kostajh - Could this be due to the same cause as the recent, similar issues for hCaptcha scripts?

I replicate error and even on translation and sections endpoints.

Removed by the incident, returning to previous state.

In T426323#11923472, @sbassett wrote:

@ssingh @kostajh - Could this be due to the same cause as the recent, similar issues for hCaptcha scripts?

Yes, this looks similar to T426178: hCaptcha: Failed to load resource: Origin (domain) is not allowed by Access-Control-Allow-Origin (although with a different stack involved, since CX is not using the nginx config that is responsible for T426178)

Change #1287731 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/deployment-charts@master] api-gateway: Add Vary: Origin to api.wikimedia.org responses

https://gerrit.wikimedia.org/r/1287731

Change #1287814 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/ContentTranslation@master] ContentTranslation: Log recommendation API errors to logstash

https://gerrit.wikimedia.org/r/1287814

Change #1287814 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] ContentTranslation: Log recommendation API errors to logstash

https://gerrit.wikimedia.org/r/1287814

From what I can tell, this behaviour changed when we moved from the old api-gateway to the rest-gateway. The envoy configuration is identical:

• Cache-Control: no-cache, which confusingly means revalidate every time, not "don't cache"
• Identical CORS header config
• No Vary: Origin header in any of the configs

What changed though is the plugin stack in ATS (edge routing) that we are now using instead of a no-plugin map. This was necessary to be able to progressively migrate traffic for T422804: Reroute LiftWing endpoints but it's very possible something in there changed caching behaviour.

Tagging Traffic for help making sense of what was the old behaviour and what we can do to restore it.

Change #1290708 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] gateway-check: Temp route linkrecommendation to api-gateway

https://gerrit.wikimedia.org/r/1290708

Change #1287731 abandoned by Kosta Harlan:

[operations/deployment-charts@master] rest-gateway: Add Vary: Origin to CORS-enabled routes

Reason:

Probably not the way to fix this

https://gerrit.wikimedia.org/r/1287731

Change #1290761 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+20260521

https://gerrit.wikimedia.org/r/1290761

Mentioned in SAL (#wikimedia-operations) [2026-05-21T14:57:19Z] <claime> Disabling puppet on A:cp-text - T426323

Change #1290708 merged by Clément Goubert:

[operations/puppet@production] gateway-check: Temp route linkrecommendation to api-gateway

https://gerrit.wikimedia.org/r/1290708

Change #1290761 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+20260521

https://gerrit.wikimedia.org/r/1290761

Mentioned in SAL (#wikimedia-operations) [2026-05-21T15:19:05Z] <claime> Enabling puppet on A:cp-text - T426323

Easy way to repro in browser:

• Open dev tools
• Go to https://fr.wikipedia.org/w/index.php?title=Special:ContentTranslation&filter-type=automatic&filter-id=previous-edits&active-list=suggestions&from=fr&to=en#/
• Various api.wikimedia.org call results get cached
• Change the language code (and only that)
• Given the params are the same, the same api.wikimedia.org calls are made, but with the new Origin
• Most api.wikimedia.org calls fail (i.e. not only /page-collection-groups)

Changing the backend in ATS changes nothing, the CORS errors still pop-up.
Testing ATS directly, routing through the rest-gateway

cgoubert@cp6013:~$ curl -o /dev/null -s -v -H 'Origin: it.wikipedia.org' -H 'Host: api.wikimedia.org' 'http://api.wikimedia.org:3128/service/lw/recommendation/api/v1/translation/sections?source=es&target=en&count=6' 2>&1 | grep -i origin
> Origin: it.wikipedia.org
< access-control-allow-origin: it.wikipedia.org
cgoubert@cp6013:~$ curl -o /dev/null -s -v -H 'Origin: en.wikipedia.org' -H 'Host: api.wikimedia.org' 'http://api.wikimedia.org:3128/service/lw/recommendation/api/v1/translation/sections?source=es&target=en&count=6' 2>&1 | grep -i origin
> Origin: en.wikipedia.org
< access-control-allow-origin: en.wikipedia.org

These are cache misses every time, meaning it's not ATS caching the wrong Access-Control-Allow-Origin, it's varnish.

Testing the rest-gateway and api-gateway directly shows both reflect Origin in Access-Control-Allow-Origin

cgoubert@deploy1003:~$ curl -o /dev/null -s -v -H 'Origin: fr.wikipedia.org' -H 'Host: api.wikimedia.org' 'https://api-gateway.discovery.wmnet:8087/service/lw/recommendation/api/v1/translation/sections?source=es&target=en&count=6' 2>&1 | grep -i origin
> Origin: fr.wikipedia.org
< access-control-allow-origin: fr.wikipedia.org
cgoubert@deploy1003:~$ curl -o /dev/null -s -v -H 'Origin: it.wikipedia.org' -H 'Host: api.wikimedia.org' 'https://rest-gateway.discovery.wmnet:4113/service/lw/recommendation/api/v1/translation/sections?source=es&target=en&count=6' 2>&1 | grep -i origin
> Origin: it.wikipedia.org
< access-control-allow-origin: it.wikipedia.org

I have to admit I don't understand how it ever worked, because as far as I can tell, reflecting the Origin in Access-Control-Allow-Origin has been envoy's (the software running in the gateways) behavior forever, and nothing obviously changed in the edge cache stack that would mean we used to override the ACAO, or not cache it.

Unless there are security justifications I'm not aware of for locking down CORS at the gateway level, the backend service returns Access-Control-Allow-Origin: '*' correctly on its own. If that's acceptable, we can remove envoy's allow_origin_string_match configuration for this group of routes and it should hopefully work.

Change #1290865 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Let recommendation-api-ng set CORS headers

https://gerrit.wikimedia.org/r/1290865

Change #1291944 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Fix cache-control header

https://gerrit.wikimedia.org/r/1291944

Change #1291944 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Fix cache-control header

https://gerrit.wikimedia.org/r/1291944

Change #1290865 abandoned by Clément Goubert:

[operations/deployment-charts@master] rest-gateway: Let recommendation-api-ng set CORS headers

Reason:

Not needed

https://gerrit.wikimedia.org/r/1290865

In T426323#11947857, @Clement_Goubert wrote:

While investigating with @hnowlan he observed the cache-control header was not being added correctly to responses coming through the rest-gateway.

Well it turns out that if your cache-control header is added to the request and not the response, it doesn't work. The setting we use, cache-control: no-cache, means "revalidate the response with the backend before serving cached content" (and not "don't cache the content at all").

Fixing that and purging https://api.wikimedia.org/service/lw/recommendation/api/v1/translation/page-collection-groups makes the CORS errors go away, mostly because the responses are now always cache misses.

I introduced this error when migrating the routes from the api-gateway to the rest-gateway and didn't catch it, my apologies.

I'm resolving this task, feel free to reopen if you disagree.

thank you for tracking this down and fixing it!