tl;dr: add a new configuration variable and related functionality allowing the MW REST API to treat arbitrary services with OpenAPI specs as REST modules.

Context

The MW REST API supports "REST modules", which are collections of endpoints sharing a version and an OpenAPI spec. REST modules can be implemented in MediaWiki core or its extensions, exposed via the /discovery endpoint, and published to the REST Sandbox. With the upcoming Audience Designations change (see T422551: REST: Audience Designations - technical implementation and its subtasks), REST modules can also have an "audience designation", such as "beta" or "internal" that defines how it is exposed and published. These default behaviors will also be able to be overridden on a per-wiki, per-module basis by the RestModuleOverrides configuration variable.

Arbitrary specs from any service can currently be published to the REST Sandbox via the RestSandboxSpecs configuration variable. (In practice, we expect to only expose specs for services hosted on WMF production, but that's a policy requirement, not a technical one.) However, arbitrary specs cannot be exposed via the /discovery endpoint, which is designed to expose modules, and generally their treatment is inconsistent with internal modules implemented in MediaWiki or its extensions.

Goal

It is desirable to make external services exposed through the MW REST system behave more consistently with internal modules. While there will always be differences, the two can be made more similar.

Implementation

External modules will be defined by a new configuration variables, RestExternalModules, which will look something like:

$wgRestExternalModules = [
    "mymodule/v1-beta": {
      "info": {
        "description": "Access to Wikimedia content and data.",
        "title": "Wikimedia REST API",
        "version": "1.0.0"
      },
      "base": "https://en.wikipedia.org/api/rest_v1",
      "spec": "https://en.wikipedia.org/api/rest_v1/?spec"
    },
]

The new config variable is keyed by module id and contains the necessary fields to expose the external module via /discovery, publish it in the REST Sandbox, expose its spec via the /module endpoint of the specs module, apply an audience designation, and control it on a per-wiki basis via RestModuleOverrides.

The array_replace_recursive merge strategy should be applied to RestExternalModules.

For internal modules, the /module endpoint returns the actual OpenAPI spec. For external modules, the /module endpoint should instead return an HTTP 30x response code (likely a 301 to facilitate caching, but the implementing engineer can confirm). We will not proxy external specs through MediaWiki.

After this change, it will no longer be possible to publish an external module to the REST Sandbox without it appearing in /discovery. (Note: that's a lie. We'll need a deprecation period for RestSandboxSpecs, and during that deprecation period it will technically still be possible. But we want to aggressively switch to the new system.)

Notes

• External modules will not have module definition files. These files are specific to the MW REST API framework, and we should not expect or require external services to know about them or provide them.
• We can't assume anything about the path structure of external module endpoints. This means we can't expect the actual endpoints/paths to follow MW REST API conventions about audience designations, module versions, or anything else. The only place we can require compliance is in the RestExternalModules config variable.
• We can and should expect an OpenAPI spec to exist for an external module, as a minimal requirement for being listed as a module.
• We can't expect OpenAPI specs for external modules to use any custom fields that we may define, comply with our linting rules, or follow any standards beyond basic OpenAPI
• An external module doesn’t necessarily have to be an actual separate piece of software. For example, the old RESTbase endpoints are now (mostly) implemented as redirects to MW REST API core endpoints, with OpenAPI specs provided as static json files in MW config. From a caller’s perspective, these endpoints can be represented as an external module.
• External modules will not be represented in MW core by an instance of any class derived from the existing "Module". That class includes functions like "findHandlerMatch()" and "execute()" which make no sense for external modules.
• If we needed to, we could introduce something like “ModuleBase” that the existing Module class derives from, and which an ExternalModule class also derives from. It is unclear if that will be necessary or helpful - likely it won't - but it would be possible.
• External modules are a display/documentation/discovery concept, not an execution one. We will not proxy invocation of external module endpoints through MediaWiki. In other words, the local wiki will not provide a /w/rest.php equivalent for endpoints in external modules.
• Invocation of external modules through the REST Sandbox "Try it out" button originates from the client browser and directly hits the external module endpoint. These will be supported.

By their nature, external modules are disconnected from MediaWiki. It is possible for details of an external module to change at any time, and for the information in config to become out of sync. For example, a service providing the endpoints referenced by an external module might change its version without changing its base or spec endpoints. MediaWiki would have no way to detect this. Config would need to be manually updated. Or the service could be offline, either temporarily or permanently, causing the REST Sandbox to fail for that external module. We should review relevant error handling to make sure we gracefully communicate such failures to users.

Future

It would be possible to also support something like a "file" key for each external module which, if present, is the url of a json file containing the necessary information. This would allow external services to keep the relevant information within their own repo, where they could maintain it directly without having to remember to and be comfortable with updating MediaWiki config.

This could be done similarly to how RestSandboxSpecs currently handles "file". Any specific keys included in addition to "file" would override any matching keys from the external file.

Nothing about this expansion changes anything with the initial implementation, so if it is desired, it can be done in a separate phase.