Page MenuHomePhabricator

Add rest-gateway routing to Beta Cluster (route PCS without RESTBase)
Closed, ResolvedPublic

Description

I think the problem might be the way MW on beta talks to RESTBase. […] Ideally since (other than serving mathoid) RESTBase is not used in production, we should replicate the same behaviour in beta cluster too.

[…] Two ways come to mind:

  1. ATS plugin + REST Gateway — This would be most like prod
    • in ATS, keep the rb-mw-mangling plugin, which rewrites part of the RESTBase URL already.
    • in ATS, add a beta version of the gateway-check.lua plugin to direct various subpaths under /api/rest_v1/ to rest-gateway
    • set up REST Gateway somewhere (Helm chart). This may be non-trivial to do in Beta. I suspect it would not actually be abl to reuse much since it seems fairly specific to production, and the indirection would presumably do very little. In prod it takes care of monitoring and in theory might do throttling, but we might not need that in Beta.
  2. ATS plugin only — This would be similar to prod but simpler.
    • in ATS, keep the rb-mw-mangling plugin, which rewrites part of the RESTBase URL already.
    • in ATS, expand the rb-mw-mangling-beta plugin to add more rewrites. This would be similar to what the gateway-check.lua plugin and rest-gateway service do together in prod. In other words, instead of rewriting /api/rest_v1/page/summary/(.*) to a rest-gateway call (which then proxies to mobileapps service), rewrite it directly to wherever the mobileapps service runs in beta.

[…]

I would recommend [the] simpler setup that should achieve the same, with less maintenance, less initial setup, and less risk of breakage. […]

My idea/patch here is a long-term improvement to solve a pre-existing problem, […]. That is volunteer work and has no timeline. […]

See also:

Event Timeline

Change #1182652 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] trafficserver: Add missing REST Gateway for Beta Cluster

https://gerrit.wikimedia.org/r/1182652

Change #1182652 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] trafficserver: Add missing REST Gateway for Beta Cluster

https://gerrit.wikimedia.org/r/1182652

Mentioned in SAL (#wikimedia-releng) [2025-09-12T16:23:42Z] <Krinkle> Enable /etc/trafficserver/lua/rb-mw-mangling-beta.lua plugin on /api/rest_v1 (restbase05) and /w/api.api (mediawiki13) regex_map routes in the Beta Cluster. It was previously only enabled for the mediawiki14 route, which ironically does not cover restbase. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/1310d67ac6dffe81e4e7a2e68ec33cbba1a7c7ea%5E%21/#F0 ref T404387

As of Patch Set 8, this appears to be working.

I cherry-picked this to the pupperserver in the Beta Cluster, and did a run-puppet-agent on the deployment-cache-text Varnish/ATS server.

Given there's heavy caching, I then cleared the Varnish and ATS caches (there wasn't a documented way to clear the ATS cache, so I brute forced something and documented it as a start). [1][2]

[1]: https://wikitech.wikimedia.org/wiki/Varnish#One-off_purges_(bans)
[2]: https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Forcing_a_cache_miss_(similar_to_ban)

Using https://en.wikipedia.beta.wmcloud.org/api/rest_v1/page/summary/Polar_bear as an example. It seems to work the same as before.

The headers confirm that it is no longer using RESTBase.

Before
HTTP/2 200 
content-type: application/json; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/Summary/1.5.0"
cache-control: s-maxage=1209600, max-age=300
content-language: en
content-location: https://en.wikipedia.beta.wmcloud.org/api/rest_v1/page/summary/Polar_bear
access-control-allow-origin: *
access-control-allow-methods: GET,HEAD
access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
access-control-expose-headers: etag
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
referrer-policy: origin-when-cross-origin
x-xss-protection: 1; mode=block
content-security-policy: default-src 'none'; frame-ancestors 'none'
x-content-security-policy: default-src 'none'; frame-ancestors 'none'
x-webkit-csp: default-src 'none'; frame-ancestors 'none'
server: deployment-restbase05
date: Fri, 12 Sep 2025 16:08:07 GMT
etag: W/"655492/6fc09540-8ef1-11f0-b283-e9f2bf0ad542"
content-encoding: gzip
age: 5285
accept-ranges: bytes
vary: x-restbase-compat, Accept-Encoding
x-cache: deployment-cache-text08 miss, deployment-cache-text08 hit/7
x-cache-status: hit-remote
server-timing: cache;desc="hit-remote", host;desc="deployment-cache-text08"
…
content-length: 581
{"type":"standard","title":"Polar bear","displaytitle":"<span class=\"mw-page-title-main\">Polar bear</span>","namespace":{"id":0,"text":""},"wikibase_item":"Q629064","titles":{"canonical":"Polar_bear","normalized":"Polar bear","display":"<span class=\"mw-page-title-main\">Polar bear</span>"},"pageid":6197,"thumbnail":{"source":"https://upload.wikimedia.org/wikipedia/commons/thumb/0/09/Polar_Bear_-_Alaska.jpg/330px-Polar_Bear_-_Alaska.jpg","width":320,"height":213},"originalimage":{"source":"https://upload.wikimedia.org/wikipedia/commons/0/09/Polar_Bear_-_Alaska.jpg","width":900,"height":600},"lang":"en","dir":"ltr","revision":"655492","tid":"260c2381-6190-11f0-a329-2631a9e0159c","timestamp":"2025-07-15T15:26:59Z","description":"Species of bear native to the Arctic","description_source":"local","content_urls":{"desktop":{"page":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear","revisions":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear?action=history","edit":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear?action=edit","talk":"https://en.wikipedia.beta.wmcloud.org/wiki/Talk:Polar_bear"},"mobile":{"page":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear","revisions":"https://en.wikipedia.beta.wmcloud.org/wiki/Special:History/Polar_bear","edit":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear?action=edit","talk":"https://en.wikipedia.beta.wmcloud.org/wiki/Talk:Polar_bear"}},"extract":"\n\nTemplate:Use Canadian English","extract_html":"<p>\n\nTemplate:Use Canadian English</p>"}
After
HTTP/2 200 
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'none'; frame-ancestors 'none'
x-content-security-policy: default-src 'none'; frame-ancestors 'none'
x-webkit-csp: default-src 'none'; frame-ancestors 'none'
cache-control: s-maxage=1209600, max-age=300
content-language: en
content-type: application/json; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/Summary/1.5.0"
date: Fri, 12 Sep 2025 17:41:33 GMT
server: ATS/9.2.11
etag: W/"655492/b9185f20-8fff-11f0-9723-f83d973083a8"
content-encoding: gzip
age: 24
accept-ranges: bytes
vary: Accept-Language, x-restbase-compat, Accept-Encoding
x-cache: deployment-cache-text08 miss, deployment-cache-text08 hit/2
x-cache-status: hit-remote
server-timing: cache;desc="hit-remote", host;desc="deployment-cache-text08"
…
content-length: 582
{"type":"standard","title":"Polar bear","displaytitle":"<span class=\"mw-page-title-main\">Polar bear</span>","namespace":{"id":0,"text":""},"wikibase_item":"Q629064","titles":{"canonical":"Polar_bear","normalized":"Polar bear","display":"<span class=\"mw-page-title-main\">Polar bear</span>"},"pageid":6197,"thumbnail":{"source":"https://upload.wikimedia.org/wikipedia/commons/thumb/0/09/Polar_Bear_-_Alaska.jpg/330px-Polar_Bear_-_Alaska.jpg","width":320,"height":213},"originalimage":{"source":"https://upload.wikimedia.org/wikipedia/commons/0/09/Polar_Bear_-_Alaska.jpg","width":900,"height":600},"lang":"en","dir":"ltr","revision":"655492","tid":"260c2383-6190-11f0-9e9b-f174a79ff2dd","timestamp":"2025-07-15T15:26:59Z","description":"Species of bear native to the Arctic","description_source":"local","content_urls":{"desktop":{"page":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear","revisions":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear?action=history","edit":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear?action=edit","talk":"https://en.wikipedia.beta.wmcloud.org/wiki/Talk:Polar_bear"},"mobile":{"page":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear","revisions":"https://en.wikipedia.beta.wmcloud.org/wiki/Special:History/Polar_bear","edit":"https://en.wikipedia.beta.wmcloud.org/wiki/Polar_bear?action=edit","talk":"https://en.wikipedia.beta.wmcloud.org/wiki/Talk:Polar_bear"}},"extract":"\n\nTemplate:Use Canadian English","extract_html":"<p>\n\nTemplate:Use Canadian English</p>"}

Note there is no server: deployment-restbase05. The JSON response is identical, except for a slightly newer UUID value in the tid field.

-  "tid":"260c2381-6190-11f0-a329-2631a9e0159c"
+  "tid":"260c2383-6190-11f0-9e9b-f174a79ff2dd"

Change #1182652 merged by Ssingh:

[operations/puppet@production] trafficserver: Add missing REST Gateway for Beta Cluster

https://gerrit.wikimedia.org/r/1182652

Krinkle triaged this task as Medium priority.