So far, the only thing I could find online is this: https://github.com/Trax-air/swagger-aggregator which seems a bit old (last commit 6 years ago).

Creating a single openAPI spec from multiple files: https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files/

In T354745#9529817, @aborrero wrote:

Creating a single openAPI spec from multiple files: https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files/

cool

I have been exploring this, with the 3 openapi definitions:

• envvars https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/blob/main/openapi/v1.yaml
• builds https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/blob/main/openapi/v1.yaml
• jobs https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/59

For a number of reasons, directly merging them will be very difficult and cumbersome:

• conflicting paths. For example, /v1/healthz or /v1/quota are defined in ennvars, builds and jobs.
• conflicting component schemas. For example, Quota is defined in builds and jobs. Envvars uses EnvvarQuota though.
• overlapping server URL specification. For example, builds and envvars use /v1, which is too generic. In jobs, it contains a more precise https://api.svc.tools.eqiad1.wikimedia.cloud:30003/jobs/api/v1.

For a merge to happen we would need first to update all 3 spec files to overcome all the conflicts, make them coherent, then write some script to do the merge.
Then, establish some kind of procedure or mechanism to ensure we don't diverge from this coherence in the future.

Maybe another, simpler approach would be to skip the unified openapi spec definition, and go directly to python lib packages:

• for each spec file, generate a python library package.
• in the repo for the consolidated client, use all the python library packages.
• this can even be done on a single repository, i.e, toolforge-api-python-libs, where we have a script that fetches each individual openapi spec, generates the python libs, and published them to pip.

Going this route would leave the question open about how/where to publish the API documentation. Well, I would just publish each individual openapi spec as they are, via wikitech.

That would also defeat the main purpose of the API unification, that is, presenting a single abstraction to the users, this is a single entry point, a single dependency, a single documentation and a single library.

All the APIs are currently served under the same url, just prefixed with the API name (ex. /build/v1/healthz), so the endpoints are already there and coexist without problems, what's left is generating the definition for them.

In T354745#9537843, @dcaro wrote:

That would also defeat the main purpose of the API unification, that is, presenting a single abstraction to the users, this is a single entry point, a single dependency, a single documentation and a single library.

This as theoretical goal is OK, but the reality is that today our architecture is a microservices one.

The challenge remains: how to bring coherence to the individual spec files (and maintain such coherence).

Maybe if we want a single library the easier would be to craft a monolitic API.

In T354745#9537865, @aborrero wrote:

In T354745#9537843, @dcaro wrote:

That would also defeat the main purpose of the API unification, that is, presenting a single abstraction to the users, this is a single entry point, a single dependency, a single documentation and a single library.

This as theoretical goal is OK, but the reality is that today our architecture is a microservices one.

The challenge remains: how to bring coherence to the individual spec files (and maintain such coherence).

Maybe if we want a single library the easier would be to craft a monolitic API.

I think we are going round and round the same discussions here (of which you were part of too), let's try to avoid it and instead to be constructive.

We can aggregate each of the APIs definitions as they are now, under the APIs subpath (jobs under /jobs/*, builds under /builds/*, etc.), that solves already the issue of the path collisions.

So the only thing left is datastructure name collisions, I can try giving that a look in a bit and come up with something. If in the end we have to change the names of those, it's a backwards compatible and simple change to do anyhow, so still good.

My concerns are also related to maintainability. In particular, if we solve by hand the collisions today, how to prevent them from sprawling in the future, across different repos.

And, in general, how to deterministically control and ensure that the individual openapi specs are crafted in a way that will play nicely later in the aggregated one.

• Apimatic is a proprietary service so we can't use it, but they have this article about microservices API integration which I found interesting: https://www.apimatic.io/blog/2022/09/auto-merging-apis-and-microservices-specifications-to-ease-api-integration.
• https://github.com/robertmassaioli/openapi-merge is a tool for merging multiple OpenAPI 3.0 files together. The last commit was from 2 years ago though
• https://github.com/iouring-engineering/openapi-merge is a fork of the above with some added functionality

Even though exposing multiple services underneath a single API Gateway is a common pattern, there doesn't seem to be much open-source tooling in this area. There are also API Gateways with integrated support for OpenAPI https://openapi.tools/#gateway. Only fusio seems to be open source though.

Doing it "on the fly", so we don't have to generate a new definition any of the sub-apis changes

@dcaro, do you mean not merging the separate OAI specs into a single file first, and then exposing it? Like, if each API exposes its own spec on an endpoint, the Gateway grabs these and generates a unified spec dynamically?

@dcaro, do you mean not merging the separate OAI specs into a single file first, and then exposing it? Like, if each API exposes its own spec on an endpoint, the Gateway grabs these and generates a unified spec dynamically?

Ideally, generating it on the fly would be better, but if it's not feasible, doing it offline with a script might be doable.

This silly script that I just started when you wrote the comment already "almost" generates a valid openapi from the envvars, builds and jobs API defs:

from __future__ import annotations
import yaml
from pathlib import Path
from typing import Any


OPENAPI_SOURCES = {
    "builds": Path("../../builds-api/openapi/v1.yaml"),
    "jobs": Path("../../jobs-api/openapi/openapi.yaml"),
    "envvars": Path("../../envvars-api/openapi/v1.yaml"),
}
BASE_API = Path("base.yaml")


def rebase_refs(api_def: dict[str, Any], old_path: str, new_path: str) -> None:
    for key, value in api_def.items():
        if key == "$ref":
            new_value = value.replace(old_path, new_path)
            api_def[key] = new_value

        else:
            if isinstance(value, dict):
                rebase_refs(api_def=value, old_path=old_path, new_path=new_path)


def rebase_api_paths(api_def: dict[str, Any], base_url: str) -> None:
    rebased_paths = {
        f"{base_url}{old_url}": data for old_url, data in api_def["paths"].items()
    }
    api_def["paths"] = rebased_paths


def main():
    merged_api = yaml.safe_load(BASE_API.open("r"))
    for api_name, source in OPENAPI_SOURCES.items():
        old_api = yaml.safe_load(source.open("r"))
        rebase_api_paths(api_def=old_api, base_url=f"/{api_name}")
        rebase_refs(
            api_def=old_api,
            old_path="/components/schemas/",
            new_path=f"/components/schemas/{api_name}",
        )
        merged_api["paths"].update(old_api["paths"])
        try:
            merged_api["components"]["schemas"].update(
                {
                    f"{api_name}{component_name}": component_value
                    for component_name, component_value in old_api["components"][
                        "schemas"
                    ].items()
                }
            )
        except Exception:
            import pdb

            pdb.set_trace()

    Path("merged_api.yaml").write_text(yaml.safe_dump(merged_api))


if __name__ == "__main__":
    main()

Fixed, this generate a valid definition:

from __future__ import annotations
import yaml
from pathlib import Path
from typing import Any


OPENAPI_SOURCES = {
    "builds": Path("../../builds-api/openapi/v1.yaml"),
    "jobs": Path("../../jobs-api/openapi/openapi.yaml"),
    "envvars": Path("../../envvars-api/openapi/v1.yaml"),
}
BASE_API = Path("base.yaml")


def rebase_refs(api_def: dict[str, Any], old_path: str, new_path: str) -> None:
    for key, value in api_def.items():
        if key == "$ref":
            new_value = value.replace(old_path, new_path)
            api_def[key] = new_value

        elif isinstance(value, list):
            for item in value:
                if isinstance(item, dict):
                    rebase_refs(api_def=item, old_path=old_path, new_path=new_path)

        else:
            if isinstance(value, dict):
                rebase_refs(api_def=value, old_path=old_path, new_path=new_path)


def rebase_api_paths(api_def: dict[str, Any], base_url: str) -> None:
    rebased_paths = {
        f"{base_url}{old_url}": data for old_url, data in api_def["paths"].items()
    }
    api_def["paths"] = rebased_paths


def main():
    merged_api = yaml.safe_load(BASE_API.open("r"))
    for api_name, source in OPENAPI_SOURCES.items():
        old_api = yaml.safe_load(source.open("r"))
        rebase_api_paths(api_def=old_api, base_url=f"/{api_name}")
        rebase_refs(
            api_def=old_api,
            old_path="/components/schemas/",
            new_path=f"/components/schemas/{api_name}",
        )
        merged_api["paths"].update(old_api["paths"])
        merged_api["components"]["schemas"].update(
            {
                f"{api_name}{component_name}": component_value
                for component_name, component_value in old_api["components"][
                    "schemas"
                ].items()
            }
        )


    Path("merged_api.yaml").write_text(yaml.safe_dump(merged_api))


if __name__ == "__main__":
    main()

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/79

move v1 to url

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/21

move v1 to url

Note that I'm using https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/79 and https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/21

Nice! I was just experimenting with https://www.npmjs.com/package/openapi-merge-cli, merging just envvars and build to begin with. It generates a valid spec out-of-the-box but I'm also visually checking it.

sounds like a more featureful version of the above script xd, we might want to avoid the extra complications though, our merging operation should be dead simple.

Though if it's stable enough might be ok to use.

In T354745#9562980, @dcaro wrote:

sounds like a more featureful version of the above script xd, we might want to avoid the extra complications though, our merging operation should be dead simple.

Though if it's stable enough might be ok to use.

It has a whole lot of source code to finally do the same as your script xd. It also only seems able to prefix components in case of conflict, not by default.

The ability to configure it using json is nice though

{
  "inputs": [
    {
      "inputFile": "./gateway.swagger.json"
    },
    {
      "inputFile": "./jira.swagger.json",
      "pathModification": {
        "stripStart": "/rest",
        "prepend": "/jira"
      },
      "operationSelection": {
        "includeTags": ["included"]
      },
      "description": {
        "append": true
      }
    },
    {
      "inputFile": "./confluence.swagger.yaml",
      "dispute": {
        "prefix": "Confluence"
      },
      "pathModification": {
        "prepend": "/confluence"
      },
      "operationSelection": {
        "excludeTags": ["excluded"]
      }
    }
  ], 
  "output": "./output.swagger.json"
}

Being js also means that if we serve it as an API, we would have to use a nodejs server.

I think that going with python/golang might be safer (python might be enough).

So I propose adding a small python http server to the api-gateway component that only handles the /openapi.json endpoint (nginx will still forward the other endpoints for now) and generates the merged API on the fly by requesting the other /*/openapi.json endpoints (maybe cached?)

We can add also serving a swagger UI if the content-type is not json or similar.

Feel free to load the config from a json/yaml whatever, I just hardcoded it in the script.

In the future, we can tweak nginx to pass to the python code endpoints that need special handling if any, or handle all of them if needed eventually.

As in:

• Current status: user -> api-gateway:nginx -> backend-apis
• Next: (/openapi.json) user -> api-gateway:nginx -> api-gateway:gateway-app | (not /openapi.json) user -> api-gateway:nginx -> backend-apis
• Possible future ({some set of urls}) user -> api-gateway:nginx -> api-gateway:gateway-app | ({some possibly empty set of urls}) user -> api-gateway:nginx -> backend-api

WDYT?

Do you want to do it yourself? (I can if you want to do something else)

In T354745#9563040, @dcaro wrote:

Being js also means that if we serve it as an API, we would have to use a nodejs server.

I think that going with python/golang might be safer (python might be enough).

Sounds good to me.

I'm not quite convinced about the "on the fly" option vs. doing merging & validation in CI each time one of the APIs OAI spec is updated, then serving this single OAI spec from the gateway via one endpoint. The on the fly option seems more complex. If we do it in CI, we would catch any merge conflicts before trying to deploy a new version of a component. Also schema caching adds more complexity, imo.

I'm not quite convinced about the "on the fly" option vs. doing merging & validation in CI each time one of the APIs OAI spec is updated, then serving this single OAI spec from the gateway via one endpoint. The on the fly option seems more complex. If we do it in CI, we would catch any merge conflicts before trying to deploy a new version of a component. Also schema caching adds more complexity, imo.

There's no merge conflicts possible, the only possibility is that the generate openapi schema is not valid.

The big advantage of doing it on the fly, is that (as long as the generated schema is syntactically valid) the exposed schema would be the current exposed APIs, otherwise, every time we modify any of the backend APIs, there will be a period where the schema does not represent the current exposed APIs until we regenerate and deploy the gateway.

We can add some basic check to the live endpoint for openapi validity if you want, and even alert through prometheus/alertmanager if it's gets borked, that would catch most of the issues (ex. https://pypi.org/project/openapi-schema-validator/).

In T354745#9563125, @dcaro wrote:

The big advantage of doing it on the fly, is that (as long as the generated schema is syntactically valid) the exposed schema would be the current exposed APIs, otherwise, every time we modify any of the backend APIs, there will be a period where the schema does not represent the current exposed APIs until we regenerate and deploy the gateway.

I'm not sure that at our scale/rate at which we modify the API schemas this makes a substantial difference, but I'm open to trying your approach.

How often would you have the gateway polling the APIs? Or would we be doing on-demand fetching with caching?

In T354745#9563229, @Slst2020 wrote:

In T354745#9563125, @dcaro wrote:

The big advantage of doing it on the fly, is that (as long as the generated schema is syntactically valid) the exposed schema would be the current exposed APIs, otherwise, every time we modify any of the backend APIs, there will be a period where the schema does not represent the current exposed APIs until we regenerate and deploy the gateway.

I'm not sure that at our scale/rate at which we modify the API schemas this makes a substantial difference, but I'm open to trying your approach.

How often would you have the gateway polling the APIs?

We can do it literally on the fly, every request that comes for the schema pulls it from the APIs and generates them (I don't think we will have much traffic there, so validity trumps performance in my mind).

In T354745#9563294, @dcaro wrote:

We can do it literally on the fly, every request that comes for the schema pulls it from the APIs and generates them (I don't think we will have much traffic there, so validity trumps performance in my mind).

If we don't care about performance, and we're choosing this approach to avoid ever serving stale schemas, then we shouldn't do any caching either.

In T354745#9563318, @Slst2020 wrote:

In T354745#9563294, @dcaro wrote:

We can do it literally on the fly, every request that comes for the schema pulls it from the APIs and generates them (I don't think we will have much traffic there, so validity trumps performance in my mind).

If we don't care about performance, and we're choosing this approach to avoid ever serving stale schemas, then we shouldn't do any caching either.

sounds good to me, if we start seeing performance issues, we can start playing with caches/async generation of the merged schema and such

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/21

move v1 to url

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/202

envvars-api: bump to 0.0.38-20240221140654-e441981f

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/24

main.go: fix base url

dcaro closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/202

envvars-api: bump to 0.0.38-20240221140654-e441981f

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/24

main.go: fix base url

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/204

envvars-api: bump to 0.0.39-20240221153320-e6de38fd

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/204

envvars-api: bump to 0.0.39-20240221153320-e6de38fd

dcaro updated https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/79

api: move the prefix to each endpoint

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/79

api: move the prefix to each endpoint

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/205

builds-api: bump to 0.0.130-20240221160838-03b2a120

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/205

builds-api: bump to 0.0.130-20240221160838-03b2a120

@dcaro what is the base.yaml in your script?

In T354745#9623453, @Slst2020 wrote:

@dcaro what is the base.yaml in your script?

:facepalm: I forgot to paste it

dcaro@urcuchillay$ cat /home/dcaro/Work/wikimedia/api-gateway/api_gateway/base.yaml
openapi: 3.1.0
info:
  title: Toolforge API
  version: 0.0.1
servers:
  - url: "api.svc.toolforge.org"

paths: {}

components:
  schemas: {}

It has just the basic API info that only the aggregated api needs.

@dcaro thanks! Another question: each openapi spec has it's own /openapi.json endpoint. We will want the gateway to expose the unified spec created by merging the files. Should we remove the individual /openapi.json endpoints from the merged spec?

In T354745#9624011, @Slst2020 wrote:

@dcaro thanks! Another question: each openapi spec has it's own /openapi.json endpoint. We will want the gateway to expose the unified spec created by merging the files. Should we remove the individual /openapi.json endpoints from the merged spec?

Good question, I'd say so yes, not sure which benefit would they have for users of the proxy (the can't really use the inner apis directly, if they can use the inner apis directly, they can access the openapi.json for the inner api directly too xd).

@dcaro How should we deal with the security top-level element. We are currently ignoring it. Maybe add it to the base.yaml?

We are also ignoring other top-level elements, I guess because we aren't currently using them. These are tags, webhooks, and externalDocs. Do we want to keep ignoring them for now?

In T354745#9625884, @Slst2020 wrote:

@dcaro How should we deal with the security top-level element. We are currently ignoring it. Maybe add it to the base.yaml?

Yep, that will be whatever we end up doing for authentication, for now we can use the same we have in the others, but we will want to put something that makes more sense later.

We are also ignoring other top-level elements, I guess because we aren't currently using them. These are tags, webhooks, and externalDocs. Do we want to keep ignoring them for now?

Any top-level elements from the internal APIs should be ignored yes, and overwritten with the base.yaml ones if there's any, or just dropped if it has not.

In the future, we might not want to remove webhooks though, we don't have any, nor plan on adding any yet, but we might want in the future, for now we can just drop it.

operationIds need to be unique. At what level do we want to deal with this? Enforce a naming convention across the individual APis such as {serviceName}_{resourceName}_{operation}? Prefixing when merging the specs?

In T354745#9626018, @Slst2020 wrote:

operationIds need to be unique. At what level do we want to deal with this? Enforce a naming convention across the individual APis such as {serviceName}_{resourceName}_{operation}? Prefixing when merging the specs?

Not really, that can be dropped if needed, it's only used for code generation I think.

If we might need it in the future, we can then prepend the value with the internal API name like Builds_<operationid> kind of thing.

In T354745#9626026, @dcaro wrote:

In T354745#9626018, @Slst2020 wrote:

operationIds need to be unique. At what level do we want to deal with this? Enforce a naming convention across the individual APis such as {serviceName}_{resourceName}_{operation}? Prefixing when merging the specs?

Not really, that can be dropped if needed, it's only used for code generation I think.

Hmm, we do want to generate the client libraries that we will use in the consolidated CLI, though. Also, I think tools that generate documentation rely on the uniqueness of operationId to correctly link operations to descriptions, parameters, and responses.

In T354745#9626052, @Slst2020 wrote:

In T354745#9626026, @dcaro wrote:

In T354745#9626018, @Slst2020 wrote:

operationIds need to be unique. At what level do we want to deal with this? Enforce a naming convention across the individual APis such as {serviceName}_{resourceName}_{operation}? Prefixing when merging the specs?

Not really, that can be dropped if needed, it's only used for code generation I think.

Hmm, we do want to generate the client libraries that we will use in the consolidated CLI, though. Also, I think tools that generate documentation rely on the uniqueness of operationId to correctly link operations to descriptions, parameters, and responses.

Let's play with it then :), can we generate the docs without operationId being present, as in if we drop it?

If not, then let's namespace it like we do with paths and schemas.

My guess is that if operationId is not there, the client will generate the name, but I have not tried.

Do we want to expose the /healthz and /metrics endpoints in the unified spec?

The tags turned out to be useful as it makes for nicer, better organized docs. Not tried it yet, but it's also possible to add a description and and a link to each "endpoint family".

In T354745#9631425, @Slst2020 wrote:

Do we want to expose the /healthz and /metrics endpoints in the unified spec?

That'd be ok yes.

The tags turned out to be useful as it makes for nicer, better organized docs.

They should be added at the sub-api level though, we should explore also what tags make sense.

I'd make that a task for later for now, we are not yet exposing the API to users.