Description

We perform end-to-end testing as part of the CI pipeline process using Mocha. For every endpoint in the MediaWiki REST API, we need to add a check that verifies the returned response matches the public API specifications. This will allow us to automatically detect, capture, and surface instances where the endpoint behavior does not match what is documented in the response body, which increases API stability, improving the MW API developer experience.

Problem statement

The main defense that we have against spec deviation is ensuring that undocumented changes don’t make it into production in the first place. We need to catch variation in the spec vs actual endpoint behavior as part of the release process as a way to ensure documentation accuracy and prevent accidental changes. By introducing spec validation as part of end-to-end testing for every endpoint, we can improve stability for our API.

Conditions of acceptance

• Reference https://phabricator.wikimedia.org/T376341 for the implementation pattern.
• Update Mocha tests to utilize the reusable methods for fetching JSON schema response and comparing it against the php response using the validation method for all 'Search' endpoints:
  • GET /search/page?q={search}
  • GET /search/title?q={search}
  • GET /search
• Perform updates that force the test to fail; verify that it will trigger a CI error.
• Verify that the test works across different HTTP responses for each endpoint.
• Demonstrate the functionality to the team.