tl;dr

• make all REST modules available and published in the REST Sandbox by default
• add a config variable to override this in cases where it is not desired

Context

We seek to implement substantial improvements to our APIs over the next year (or more). This includes introducing new standardized patterns for how to build and structure API experiences, introducing new capabilities through API interfaces, and updating existing endpoints to follow the new structures. In many cases, these changes will be relatively high risk, with the expectation that they must remain stable once officially released.

We therefore need to create repeatable mechanisms that will enable easier and more consistent API experimentation. We propose code changes and conventions for "Beta" modules as the first step in achieving this. The overall plan for experimentation support includes changes to module paths and versions, testing patterns, and module registration/discoverability/configuration.

Today, experimentation is done in a haphazard manner across a limited set of teams. In many cases, the experimental state is only stated in documentation, which leads to some adoption in production applications.

Currently, making a new REST API module in core available to callers and visible in the REST Sandbox requires changes to two different configuration variables, and an associated config deploy. These extra steps unnecessarily complicate introduction and removal of experimental modules.

Description

Activating core modules via config is inconvenient and requires either deploy privileges (and the ability/confidence to use it), or scheduling for a deploy window. We've allowed activating extension endpoints/modules for some time without a config change, so there seems to be no process/policy/procedure reason to require it - it is just a consequence of our initial implementation.

Also, the current system requires core developers who wish to both activate their module and publish it to the REST Sandbox to modify two different config variables of different format. And extension developers have to do a config deploy to publish their modules in the REST Sandbox, which seems inconsistent and unnecessary - why should providing documentation be gated through a config deploy, when making the endpoint available in the first place is not?

However, for all its inconvenience, config is useful and powerful. The ability to control whether or not a module is enabled/published on a per-wiki basis is needed for WMF production (and is already being used today). Generally speaking, the system should make it easy to do the normal thing, while still allowing doing less common things when needed. It is okay if less common things require a reasonable amount of additional effort/inconvenience.

This leads to the following:

• endpoints for all modules (both experimental and non-experimental) are by default available and published in the REST Sandbox
• modules can be hidden from the REST Sandbox or disabled (their endpoints made unavailable) by a config change
• the config change to disable/hide a module does not depend on the existence of that module (so that you can deploy the config first, then the code, thereby avoiding a race situation where the endpoints are briefly present)

Implementation

The above has two notable consequences:

• enabling modules by default means that core modules must be automatically discoverable
• we don't already have a config variable that would allow disabling an otherwise-available module, or hiding a module from the REST Sandbox that would otherwise appear there. So changes to our config approach will be necessary

There are are several ways we could go about making core modules discoverable, including:

1. move module definition files into a specific directory, which core will scan at execution time (and cache the results)
2. introduce a listing file, probably json, that points at module definition files
3. use config defaults for core, given that core already knows about its own endpoints

#1 is simple for endpoint developers, and makes it easy to see all core module definition files (right now, they're mixed with other files in includes/Rest, which has always felt messy to the author of this task).
#2 feels like configuration within code, adds another thing for developers to remember to do, and introduces a new schema definition.
#3 is literally configuration within code, and adds a non-obvious thing for developers to do outside the Rest directory - modifying config defaults when making a code change isn't a typical pattern

We will therefore do #1. Efficient caching is essential.

For developers who need to disable/hide modules on a per-wiki basis, introduce a new config variables, "RestModuleOverride". This allows overriding the values detected from the module definition file. For example:

$wgRestModuleOverride['mymodule.v1-beta.json'] = [
	'enable' => false,
	'publish' => false
];

The existing config variables are still needed, at least for now. wgRestAPIAdditionalRouteFiles will continue to be used for includes/Rest/coreDevelopmentRoutes.json. wgRestSandboxSpecs will be useful for the default module (until all endpoints are in modules and the legacy ones are removed), but will also continue to be useful if we want to point the Rest Sandbox at things like services outside MediaWiki.

The code should refuse to (attempt to) publish a module that is not enabled. This could be logged at a non-intrusive level, to help identify the erroneous config, but should not display any user-visible error or create excessive logspam.

Future

We also discussed a checkbox allowing the user to choose whether or not they see experimental modules in the REST Sandbox. Or possibly just marking experimental modules in the selection list, maybe by appending "(Experimental)". This would require recognizing experimental modules, presumably the presence of a suffix. And maybe also v0.

None of this is necessary at this time. Given anticipated changes to the REST Sandbox implementation (Swagger UI => Codex), it is difficult to predict the best way that we would go about this. The implementation therefore does not need to make allowences for this.