Page MenuHomePhabricator

Expose LiftWing API for serving the openapi-specs through the /docs yaml files.
Closed, ResolvedPublic

Description

Scope

We would like to build an API in inference-services repository which exposes the openapi-specs (yaml files) for each one of the LiftWing models. This way we can configure the RestSandbox and make these v1 endpoints available for testing via the swagger-ui (using the "TryOut" option in the UI).
Due to limitations of the out-of-the-box Kserve specs (more info in this comment), we need to expose them in a more custom way.
This API will not be a classic Kserve model-server similar to our other models we need to understand how this can be deployed as a service.

Requirements
  • LiftWing API exposing the /docs
  • Deployment
  • Endpoint url

Please feel free to add more information if you find something unclear or if there are any missing parts.

Details

Other Assignee
Clement_Goubert
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Gkyziridisoperations/deployment-chartsmaster+1 -1
Clément Goubertoperations/deployment-chartsmaster+2 -2
Clément Goubertoperations/deployment-chartsmaster+4 -1
Gkyziridisoperations/mediawiki-configmaster+16 -0
Catropeoperations/mediawiki-configmaster+0 -16
Gkyziridisoperations/mediawiki-configmaster+16 -0
Gkyziridismachinelearning/liftwing/inference-servicesmain+76 -5
Gkyziridisoperations/deployment-chartsmaster+1 -1
Clément Goubertoperations/deployment-chartsmaster+3 -6
Gkyziridisoperations/mediawiki-configmaster+10 -2
Clément Goubertoperations/deployment-chartsmaster+1 -0
Clément Goubertoperations/deployment-chartsmaster+25 -0
Gkyziridisintegration/configmaster+1 -0
Clément Goubertoperations/deployment-chartsmaster+1 -1
Clément Goubertoperations/puppetproduction+1 -0
Gkyziridisoperations/dnsmaster+9 -6
Clément Goubertoperations/puppetproduction+13 -0
Gkyziridisoperations/deployment-chartsmaster+118 -5
Gkyziridismachinelearning/liftwing/inference-servicesmain+125 -11
Gkyziridisoperations/puppetproduction+14 -0
Gkyziridisintegration/configmaster+15 -0
Ilias Sarantopoulosmachinelearning/liftwing/inference-servicesmain+29 -0
Gkyziridismachinelearning/liftwing/inference-servicesmain+24 -1
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1296565 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] feat(openapi-specs): Expose edit-check spec from the model server

https://gerrit.wikimedia.org/r/1296565

Change #1296595 had a related patch set uploaded (by Gkyziridis; author: AikoChou):

[machinelearning/liftwing/inference-services@main] feat(openapi-specs): Expose edit-check spec from the model server

https://gerrit.wikimedia.org/r/1296595

Change #1296595 abandoned by Gkyziridis:

[machinelearning/liftwing/inference-services@main] feat(openapi-specs): Expose edit-check spec from the model server

Reason:

Wrong authors

https://gerrit.wikimedia.org/r/1296595

Change #1296595 restored by Gkyziridis:

[machinelearning/liftwing/inference-services@main] feat(openapi-specs): Expose edit-check spec from the model server

https://gerrit.wikimedia.org/r/1296595

Change #1296595 abandoned by Gkyziridis:

[machinelearning/liftwing/inference-services@main] feat(openapi-specs): Expose edit-check spec from the model server

https://gerrit.wikimedia.org/r/1296595

Thanks for working on this, @gkyziridis! The end goal should be to have one endpoint that serves the YAML for all endpoints. For example: https://api.wikimedia.org/service/lw/inference/v1/openapi.yaml

Since the inference-services repository stores the OpenAPI docs as one file per endpoint, we'll need to find a way to combine the files into one file before it is served. That way the docs can appear in the REST Sandbox as a unified Lift Wing API. If we were to serve the OpenAPI files only as separate endpoints, they would appear in the sandbox as one API per endpoint, which would break the pattern that the sandbox uses.

Thnx for your comments @apaskulin!
I was investigating @Clement_Goubert's option for exposing the openapi specs on the Kserve level by overriding the standard specs with the custom ones.
This works fine I tested it locally configuring it on the RestSandbox.

Indeed the goal is to have a single option called: LiftWing APIs which will show all the endpoints. In order to do that we need somehow to find a way to aggregate all the individually exposed specs from each model under an umbrella, an then expose this one, which leads us again back to the issues:

  1. "how will serve the single file openapi.yaml which $refs all the individual models?" or
  2. "how we will aggregate on the top level all individual openapi specs which are exposed on the Kserve level?"

Number 1. drives us to create a dedicated endpoint which serves the /docs folder, this will follow the pattern of the sandbox.
I am investigating number 2 at the moment trying to find a way to aggregate them somehow. The closest thing I achieved is depicted in the following picture:

RestSandbox LiftWing group (1,600×350 px, 45 KB)

Change #1297070 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[integration/config@master] inference-services: Add liftwing-openapi-server CI/CD pipelines.

https://gerrit.wikimedia.org/r/1297070

Change #1297072 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] feat(liftwing-openapi-server): Add static OpenAPI spec server with CORS support

https://gerrit.wikimedia.org/r/1297072

We decided to go with the option 1, having a dedicated endpoint liftwing-openapi-server which serves the umbrella yaml for all endpoints. I've tested it locally and it works fine with the RestSandbox.

LiftWing configure on RestSandbox (2,916×1,854 px, 462 KB)

I've opened the above patches one for building the LiftWing endpoint for serving the docs, and one for configuring the CI/CD Jobs.
What I miss, is how we can proceed on a deployment of such an API which is not Kserve based.

Change #1297070 merged by jenkins-bot:

[integration/config@master] inference-services: Add liftwing-openapi-server CI/CD pipelines.

https://gerrit.wikimedia.org/r/1297070

You can probably use the python-webapp chart like for recommendation-api-ng

+1 . on the deployment-side we already have the capabilities and prior art of deploying python based apps (like rec api ng and ores-legacy) .
@gkyziridis I'd recommend to use ores-legacy which is similar to what we are trying to do here (repo is inference-services+ uses python-webapp chart). I'm referring to helmfile.d/ml-services/ores-legacy in deployment-charts repo unless somebody has a simpler or just alternative suggestion

You can probably use the python-webapp chart like for recommendation-api-ng

+1 . on the deployment-side we already have the capabilities and prior art of deploying python based apps (like rec api ng and ores-legacy)

I've added a comment on the CR as we already have a standard way to expose static files using Apache rather than python's http.server. I will copy it here for clarity's sake:

I think the best way to expose an HTTP server like this is to adapt the pipeline that already exists for miscweb see static-codereview for instance.
That will build a simple, standard apache + your files image that you can then usie the miscweb chart and a new service similar to this with a stripped down helmfile.yaml in your cluster.

If we don't plan on doing anything fancy like rewrite the yaml on the fly, I think that's a better way to go than the python app. Happy to assist if you need any help with how to write and deploy that.

This sounds even better (and simpler)!

Thank you both @isarantopoulos and @Clement_Goubert for you help.
So just to make sure that I've understood what you suggest:

  1. Create a repo in gitlab under: repos/sre/miscweb/liftwing-openapi-specs
  2. Add the /docs files in there and configure the pipeline similar to static-codereview repo.
  3. Create a new helmfile in helmfile.d/services/miscweb for the deployment.

If the above seem correct, then I think I need some permissions and access.
I cannot create a new repo under repos/sre/miscweb/

No, no need. The goal is to have that service in the ml-services cluster, and to not have you create yet another git repo. To do so, in 1297072:

  • Change .pipeline/liftwing_openapi_server/blubber.yaml to look like this one, so it copies your docs files from the repo in the image
  • Copy and modify httpd.conf and statictendril.conf (rename to liftwingspec.conf or similar) to match, you want to change the paths in there to match, ServerAlias api.wikimedia.org and the discovery dns name
  • Copy over the entrypoint.sh as well

Then it's a standard new service deployment in your cluster (see wt:Kubernetes/Add_a_new_service):

  • Add the correct admin_ng and hieradata/common/profile/kubernetes/deployment_server.yaml configs for a new namespace
  • Add a new service in your cluster helmfile.d/ml-services/openapi-specs
  • Give it a similar (but limited to one release) helmfile.yaml as helmfile.d/services/miscweb, a corresponding values.yaml
  • Create the discovery/mesh config in hieradata/common/service.yaml hieradata/common/profile/services_proxy/envoy.yaml (see wt:Kubernetes/Ingress#Add_a_new_service_under_Ingress)

Change #1297167 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: add liftwing-openapi-server deployment

https://gerrit.wikimedia.org/r/1297167

Change #1297168 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/puppet@production] liftwing-openapi-server: Add new admin_ng service for serving OpenAPI specs

https://gerrit.wikimedia.org/r/1297168

Change #1297710 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/dns@master] dns: Add liftwing-openapi-server CNAME records

https://gerrit.wikimedia.org/r/1297710

@Clement_Goubert thank for your comments, they were super informative and helpful.
All the necessary actions are already taken and the corresponding patches are already under review.
The plan is to start deployments Monday next week following this order:

  1. Merge and publish the image of liftwing-openapi-server in registry: patch-inference-services
  2. Kubeconfig files (+mesh that could be done later but is not a problem to do right there): patch-puppet
  3. Deployment both the admin_ng part and the actual service: patch-deployment-charts
  4. Ingress configuration and DNS: patch-operations/DNS
  5. LVS in service_setup, configuring the liftwing-openapi-server in "hieradata/common/service.yaml"  in a separated puppet patch

We just have some questions (mainly for our understanding) on number 4, regarding the publicity of the endpoint and the CNAME configuration:

  • Does the endpoint actually needs to be public having a specific CNAME?
  • Can RestSandbox just query the liftwing-openapi-server endpoint internally by reaching LiftWing directly?
  • If public access is needed, would it be something that we could do via the rest-gateway?

Tagging here @HCoplin-WMF for visibility.

@Clement_Goubert thank for your comments, they were super informative and helpful.

No problem, happy to help!
[...]

We just have some questions (mainly for our understanding) on number 4, regarding the publicity of the endpoint and the CNAME configuration:

  • Does the endpoint actually needs to be public having a specific CNAME?

The CNAME is internal and gives you liftwing-openapi-server.discovery.wmnet.

  • Can RestSandbox just query the liftwing-openapi-server endpoint internally by reaching LiftWing directly?

As far as I can tell, the RestSandbox can load a server-side file, or does a client-side GET for url-defined specs, so the end-state would be to have something like https://api.wikimedia.org/service/lw/openapi.yaml or similar in wgRestSandboxSpecs

  • If public access is needed, would it be something that we could do via the rest-gateway?

Yes, we would add a specific route for it in the rest-gateway that would point to the liftwing-openapi-server listener that you defined in the puppet patch. Happy to assist in setting that up as well.

As far as I can tell, the RestSandbox can load a server-side file, or does a client-side GET for url-defined specs, so the end-state would be to have something like https://api.wikimedia.org/service/lw/openapi.yaml or similar in wgRestSandboxSpecs

The Rest Sandbox will perform a client-side request from the end user's browser to retrieve the spec.

You may have seen a 'file' option in the RestSandboxSpecs config variable, but that's for REST module definition file, not the OpenAPI spec, and doesn't help us here.

Change #1297168 merged by Dpogorzelski:

[operations/puppet@production] liftwing-openapi-server: Add new admin_ng service for serving OpenAPI specs

https://gerrit.wikimedia.org/r/1297168

Change #1297072 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] feat(liftwing-openapi-server): Serve OpenAPI specs via Apache httpd

https://gerrit.wikimedia.org/r/1297072

Change #1297167 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add liftwing-openapi-server deployment

https://gerrit.wikimedia.org/r/1297167

Change #1298779 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] service::catalog: Add liftwing-openapi-server

https://gerrit.wikimedia.org/r/1298779

Change #1298779 merged by Clément Goubert:

[operations/puppet@production] service::catalog: Add liftwing-openapi-server

https://gerrit.wikimedia.org/r/1298779

Change #1297710 merged by Clément Goubert:

[operations/dns@master] dns: Add liftwing-openapi-server CNAME records

https://gerrit.wikimedia.org/r/1297710

Change #1298794 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] admin_ng: Disable istio injection for openapi-server

https://gerrit.wikimedia.org/r/1298794

Change #1298798 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] conftool-data: Add liftwing-openapi-server

https://gerrit.wikimedia.org/r/1298798

Change #1298798 merged by Clément Goubert:

[operations/puppet@production] conftool-data: Add liftwing-openapi-server

https://gerrit.wikimedia.org/r/1298798

Change #1298804 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[integration/config@master] inference-services: Add liftwing-openapi-server CI/CD pipelines.

https://gerrit.wikimedia.org/r/1298804

Change #1298794 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Disable istio injection for openapi-server

https://gerrit.wikimedia.org/r/1298794

Change #1298819 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Add routing for liftwing-openapi-server

https://gerrit.wikimedia.org/r/1298819

Change #1298804 merged by jenkins-bot:

[integration/config@master] inference-services: Add liftwing-openapi-server CI/CD pipelines.

https://gerrit.wikimedia.org/r/1298804

Change #1298819 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Add routing for liftwing-openapi-server

https://gerrit.wikimedia.org/r/1298819

Change #1299439 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: fix lw-openapi-server host

https://gerrit.wikimedia.org/r/1299439

Change #1299439 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: fix lw-openapi-server host

https://gerrit.wikimedia.org/r/1299439

The specs file is now available externally at https://api.wikimedia.org/service/lw/specs/openapi.yaml

You can proceed with a new mediawiki-config patch for wgRestSandboxSpecs

Change #1299454 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/mediawiki-config@master] wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org

https://gerrit.wikimedia.org/r/1299454

Change #1299454 merged by jenkins-bot:

[operations/mediawiki-config@master] wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org

https://gerrit.wikimedia.org/r/1299454

Mentioned in SAL (#wikimedia-operations) [2026-06-09T20:10:12Z] <cscott@deploy1003> Started scap sync-world: Backport for [[gerrit:1299454|wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-09T20:12:16Z] <cscott@deploy1003> cscott, gkyziridis: Backport for [[gerrit:1299454|wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-06-09T20:32:20Z] <cscott@deploy1003> Finished scap sync-world: Backport for [[gerrit:1299454|wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s)

https://test.wikipedia.org/w/index.php?api=lift-wing&title=Special%3ARestSandbox is now live on Test Wiki! 🎉 I've tested it, and all the information appears as expected and "Try it out" works for all endpoints.

It is a bit slow since it is loading the endpoints from the other individual yaml files, especially when linking to a specific endpoint (example). For performance and for the ability to use the OpenAPI Linter, it would be ideal to combine the yaml files into a single file before serving. However, I don't think this needs to be a blocker to getting this into production. Another option would be to combine into a single file in the source repository.

I think we're ready to create the config patch to add Lift Wing to the REST Sandbox in production. After talking to Halley, we would like to add it to Meta (metawiki), mediawiki.org (mediawikiwiki), and Wikitech (labswiki) to cover the three main documentation wikis. If you can create the patch @gkyziridis, I can take it to the afternoon backport window again.

To fully complete this work, we'll also need to merge (and deploy):

Thanks everyone for your hard work on this!

Change #1300065 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Cache liftwing-openapi-specs

https://gerrit.wikimedia.org/r/1300065

Change #1300073 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/mediawiki-config@master] wgRestSandboxSpecs: Add Lift Wing API to documentation wikis

https://gerrit.wikimedia.org/r/1300073

Change #1300065 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Cache liftwing-openapi-specs

https://gerrit.wikimedia.org/r/1300065

I've changed the policy so that we cache the spec files, and removed the rest-gateway CORS override given the backend service sets ACA* headers. This *may* cause ACAO header issues when used from multiple different wikis, please ping me when it's deployed to documentation wikis so I can test there's nothing weird going on.

Change #1300089 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: add liftwing-openapi-server latest version deployment

https://gerrit.wikimedia.org/r/1300089

Change #1300089 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add liftwing-openapi-server latest version deployment

https://gerrit.wikimedia.org/r/1300089

The latest version of liftwing-openapi-server is deployed and it is live at: https://test.wikipedia.org/w/index.php?api=lift-wing&title=Special%3ARestSandbox

  • Includes the revscoring models and ores models
  • The models are grouped using tags in the openapi specs.

RestSandbox LiftWing group (1,482×1,874 px, 408 KB)

Change #1300108 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] docs(liftwing-openapi-server): Document architecture, local testing, and deployment

https://gerrit.wikimedia.org/r/1300108

Change #1300108 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] docs(liftwing-openapi-server): Document architecture, local testing, and deployment

https://gerrit.wikimedia.org/r/1300108

Change #1300073 merged by jenkins-bot:

[operations/mediawiki-config@master] wgRestSandboxSpecs: Add Lift Wing API to documentation wikis

https://gerrit.wikimedia.org/r/1300073

Mentioned in SAL (#wikimedia-operations) [2026-06-10T20:32:27Z] <catrope@deploy1003> Started scap sync-world: Backport for [[gerrit:1300073|wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-10T20:34:35Z] <catrope@deploy1003> catrope, gkyziridis: Backport for [[gerrit:1300073|wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-06-10T20:44:23Z] <catrope@deploy1003> Finished scap sync-world: Backport for [[gerrit:1300073|wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 11m 55s)

Change #1300247 had a related patch set uploaded (by Catrope; author: Catrope):

[operations/mediawiki-config@master] Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis"

https://gerrit.wikimedia.org/r/1300247

Change #1300247 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis"

https://gerrit.wikimedia.org/r/1300247

Mentioned in SAL (#wikimedia-operations) [2026-06-10T20:55:24Z] <catrope@deploy1003> Started scap sync-world: Backport for [[gerrit:1300247|Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]]

@Clement_Goubert We tried to deploy the new patch this afternoon but got a CORS error:

Access to fetch at 'https://api.wikimedia.org/service/lw/specs/openapi.yaml' from origin 'https://www.mediawiki.org' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://test.wikipedia.org' that is not equal to the supplied origin. Have the server send the header with a valid value.

We rolled back the patch for now. Do you have a fix you would be able to try tomorrow?

Mentioned in SAL (#wikimedia-operations) [2026-06-10T20:57:29Z] <catrope@deploy1003> catrope: Backport for [[gerrit:1300247|Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

(For completeness, the above error was at https://www.mediawiki.org/w/index.php?api=lift-wing&title=Special%3ARestSandbox , so the problem was that we were hitting the API from www.mediawiki.org but the ACAO header was for test.wikipedia.org)

Mentioned in SAL (#wikimedia-operations) [2026-06-10T21:02:15Z] <catrope@deploy1003> Finished scap sync-world: Backport for [[gerrit:1300247|Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] (duration: 06m 51s)

Change #1300731 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/mediawiki-config@master] wgRestSandboxSpecs: Add Lift Wing API to documentation wikis

https://gerrit.wikimedia.org/r/1300731

Change #1300754 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Restore no-cache for lw-openapi-server

https://gerrit.wikimedia.org/r/1300754

Change #1300731 merged by jenkins-bot:

[operations/mediawiki-config@master] wgRestSandboxSpecs: Add Lift Wing API to documentation wikis

https://gerrit.wikimedia.org/r/1300731

Mentioned in SAL (#wikimedia-operations) [2026-06-11T13:04:48Z] <gkyziridis@deploy1003> Started scap sync-world: Backport for [[gerrit:1300731|wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-11T13:06:56Z] <gkyziridis@deploy1003> gkyziridis: Backport for [[gerrit:1300731|wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Change #1300754 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Restore no-cache for lw-openapi-server

https://gerrit.wikimedia.org/r/1300754

Mentioned in SAL (#wikimedia-operations) [2026-06-11T13:13:35Z] <gkyziridis@deploy1003> Finished scap sync-world: Backport for [[gerrit:1300731|wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 08m 47s)

Change #1300797 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Vary lw-openapi-server cache on Origin

https://gerrit.wikimedia.org/r/1300797

We redeployed the mediawiki-config and it works great for mediawiki: https://www.mediawiki.org/w/index.php?api=lift-wing&title=Special%3ARestSandbox
Thank you all for your help!

Change #1300797 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Vary lw-openapi-server cache on Origin

https://gerrit.wikimedia.org/r/1300797

(For completeness, the above error was at https://www.mediawiki.org/w/index.php?api=lift-wing&title=Special%3ARestSandbox , so the problem was that we were hitting the API from www.mediawiki.org but the ACAO header was for test.wikipedia.org)

Yeah, it's basically the same issue as T426323: CX dashboard can't load page collections on some wikis (blocked by CORS)

envoy has this behaviour where if you add CORS handling and accept Access-Control-Allow-Origin: *, instead of just passing it through, it will reflect your Origin instead.

So say you're loading the REST Sandbox from meta.wimedia.org, it requests the spec file from api.wikimedia.org, the backend server says Access-Control-Allow-Origin: *, then envoy overrides that on the response with Access-Control-Allow-Origin: meta.wikimedia.org.
That gets cached entirely, body and headers, if there's no cache-control header saying otherwise. That means that if you don't add Vary: Origin to split the cache on Origin, you end up serving a cache hit including Access-Control-Allow-Origin: meta.wikimedia.org when you use the REST Sandbox through www.mediawiki.org or any other Origin.

I sill have a small issue to dig into, which is that when using the X-Wikimedia-Debug header or the WikimediaDebug extension, requests to api.wikimedia.org seem to get routed directly to mw-debug API servers without going through the REST Gateway, so the REST Sandbox is broken. This is a wider and potentially more complex problem, but is not as impactful and should not be considered a blocker. I will create a separate task for it.

Clement_Goubert updated Other Assignee, added: Clement_Goubert.

Change #1300806 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: add liftwing-openapi-server latest version deployment

https://gerrit.wikimedia.org/r/1300806

Thanks for the updates and all the work everyone!

@Clement_Goubert I'm still seeing CORS errors in the sandbox on mediawiki.org, even after clearing browser cache, trying different browsers and on mobile, and making sure Wikimedia Debug is off. It fetches https://api.wikimedia.org/service/lw/specs/openapi.yaml fine but then fails on the other yaml files (like
https://api.wikimedia.org/service/lw/specs/ores/item-topic.yaml) with the same error that we saw yesterday:

Access to fetch at 'https://api.wikimedia.org/service/lw/specs/ores/item-topic.yaml' from origin 'https://www.mediawiki.org' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://test.wikipedia.org' that is not equal to the supplied origin. Have the server send the header with a valid value.

However, it works fine for me on Meta and Wikitech, so I assume this is just a cache issue. I'll keep trying and assume it will resolve itself, but I wanted to flag it here just in case.

Thanks for the updates and all the work everyone!

@Clement_Goubert I'm still seeing CORS errors in the sandbox on mediawiki.org, even after clearing browser cache, trying different browsers and on mobile, and making sure Wikimedia Debug is off. It fetches https://api.wikimedia.org/service/lw/specs/openapi.yaml fine but then fails on the other yaml files (like
https://api.wikimedia.org/service/lw/specs/ores/item-topic.yaml) with the same error that we saw yesterday:

Access to fetch at 'https://api.wikimedia.org/service/lw/specs/ores/item-topic.yaml' from origin 'https://www.mediawiki.org' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://test.wikipedia.org' that is not equal to the supplied origin. Have the server send the header with a valid value.

However, it works fine for me on Meta and Wikitech, so I assume this is just a cache issue. I'll keep trying and assume it will resolve itself, but I wanted to flag it here just in case.

That all works for me so I assume it's the edge cache you hit. I've sent a cache purge for all the liftwing spec file URLs, so if you can wait a couple of minutes and try again please?

Everything is working now for me! Thank you!

Change #1300806 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add liftwing-openapi-server latest version deployment

https://gerrit.wikimedia.org/r/1300806