tl;dr

• Allow paths like http://localhost:8080/w/rest.php/mymodule/v0-beta/echo and http://localhost:8080/w/rest.php/mymodule/v1-beta/echo
• Allow module versions like 0.1.0-beta and 1.0.0-beta

Context

We seek to implement substantial improvements to our APIs over the next year (or more). This includes introducing new standardized patterns for how to build and structure API experiences, introducing new capabilities through API interfaces, and updating existing endpoints to follow the new structures. In many cases, these changes will be relatively high risk, with the expectation that they must remain stable once officially released.

We therefore need to create repeatable mechanisms that will enable easier and more consistent API experimentation. We propose code changes and conventions for "Beta" modules as the first step in achieving this. The overall plan for experimentation support includes changes to module paths and versions, testing patterns, and module registration/discoverability/configuration.

Today, experimentation is done in a haphazard manner across a limited set of teams. In many cases, the experimental state is only stated in documentation, which leads to some adoption in production applications.

There is also a pattern where v0 indicates that an endpoint is 'unstable'. The existing 'unstable' pattern indicates that the endpoint may change over time, but does not make guarantees for when or how frequently. This can result in unexpected breaking changes when unstable endpoints are used in production applications, which is a frequent occurrence. The proposed changes send sends a stronger signal about the frequency of change and limited lifespan of the endpoint. Specifically, beta endpoints will change frequently during the beta period as we respond to feedback, and may be either deprecated or elevated to an official version after the beta period is over. Once elevated to a stable version, they will be locked into the stable version policy.

Description

Module ids appear in url paths. For example, in the path rest.php/specs/v0/discovery, the specs/v0 portion is the module id (including version) and the discovery portion is the endpoint being called. This particular endpoint (which is used for discovering OpenAPI specs) does not have path or query parameters. but if it did they would appear after discovery.

The version portion of the module id string, in this case v0, indicates this endpoint is unstable. However, historically v0 endpoints have been used in production applications, so this indicator has lost some of its meaning. Also, there is no way to indicate that a module contains experimental endpoints for transitioning from v1 to v2. The system simply doesn't have a way to represent experiments on existing production endpoints, which is part of the reason why most endpoints stick forever at v1, even when changes are made to them.

Similarly, module version strings (within the module definition file) are often of the form 0.1 or 1.0. This does not follow semantic versioning principles, and gives no good way to indicate experimental version of modules that have already reached production.

Semantic Versioning 2.0.0 says:

A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.--.

This is quite flexible, so we'll still want our own conventions. But the key point is that prerelease versions are identified by an appended string.

We will allow both module ids and module versions to have suffixes indicating that a particular module is "beta". Also, require the initial portion of all module versions to be of the form x.y.z, fully following semantic versioning.

Example module ids:

• mymodule.v0
• mymodule.v0-beta
• mymodule.v1-beta
• mymodule.v1
• mymodule.v2-beta
• mymodule.v2

Example module versions:

• 0.1.0
• 0.1.1
• 0.1.0-beta
• 1.0.0-beta
• 1.0.0
• 1.0.1
• 1.1.0
• 2.0.0-beta
• 2.0.0
• 2.0.1
• 2.1.0

This means that (using an example from my local dev) the url of a beta module named mymodule with an endpoint named echo would look like:

• http://localhost:8080/w/rest.php/mymodule/v0-beta/echo

Compliance to these conventions should be enforced via a structure test.

Implementation Notes

From a technical point of view, a module is (currently) identified by the unique combination of name and version. In other words, from the perspective of the MW REST API infrastructure code, there is no concept that mymodule.v1 and mymodule.v2 are different versions of the same module. They're just two different strings that identify two different modules. This works to our advantage here, as there's less technical baggage.

Module names are currently expected to follow a certain pattern, specifically:

private const PREFIX_PATTERN = '!^/([-_.\w]+(?:/v\d+)?)(/.*)$!';

This essentially says that, to be recognized as a module for the purpose of matching a path to a handler via module definition files, this portion of the path must look like:

something/vnumber

Where "something" is an arbitrary string and "number" is digits. So really, the only requirement on module names right now is that a module name portion exists, and the version portion is a "v" followed by some numbers.

This is checked in the Router class, and allows the MW REST API infrastructure to identify that a particular path belongs to a module (as opposed to being a legacy route from before the module system existed).

We tend to call modules things like mymodule.v1, but in the path and the moduleId in the module definition file we use a forward slash rather than a period to separate the name and version portions. So mymodule.v1 appears in paths and module definition files as mymodule/v1. The module definition file names use the period form (because slashes in file names are problematic). For example, the file content.v1.json contains a moduleId of content/v1.

Module definition files include a version key (within the info block, required by the OpenAPI spec for info blocks). It will show up in the REST Sandbox in a small bubble by the module name. It already accepts strings, so no technical changes are needed to allow suffixes. However, as noted above, a structure test should enforce compliance with conventions, rather than allowing arbitrary strings.

Future Expansion

It might be reasonable to eventually allow other suffixes, such as alpha and rc. Examples:

Future module id possibilities:

• mymodule.v0-alpha
• mymodule.v0-rc1
• mymodule.v0-rc2
• mymodule.v1-alpha
• mymodule.v1-rc1
• mymodule.v1-rc2
• mymodule.v2-alpha
• mymodule.v2-rc1
• mymodule.v2-rc2

Future module version possibilities:

• 0.1.0-alpha
• 0.1.0-rc1
• 0.1.0-rc2
• 1.0.0-alpha
• 1.0.0-rc1
• 1.0.0-rc2
• 2.0.0-alpha
• 2.0.0-rc1
• 2.0.0-rc2

It is prudent to initially support only beta as a suffix until we see how all this works in practice and is received by developers. It may be that beta is sufficient and supporting other suffixes would add unnecessary complexity. Or it may be that other changes need to be made to the system, and having other suffixes at this time would complicate making adjustments.

The changes made under this task should support only beta. However, the changes should be implemented to easily allow other suffixes in the future if we decide to.

Also, part of the motivation for enforcing semantic versioning in the version string in module definition files is to support things outside the scope of this task. For example, automated changelogs or documentation. All that's a lot simpler if the version strings are consistent, and requiring developers to add .0 to their module versions, seems a small ask.