Page MenuHomePhabricator

Create schema validator method
Closed, ResolvedPublic

Description

Description

We will perform spec validation against actual endpoint behavior as part of the CI pipeline process. To do that, we will use a reusable Javascript method to compare the response bodies of the defined spec against the actual endpoint.

Problem statement

We need a consistent way to validate response bodies as part of end-to-end endpoint testing. Although we will still need to call this method from within end-to-end testing for every endpoint, creating a reusable validation method will allow us to maintain the logic from a single location instead of writing bespoke validation checks for each endpoint.

Conditions of acceptance

[] Implement a reusable method that will accept two versions of the responses and confirm whether they match or not.
[] If they do not match, return helpful information about what failed. Examples include:
. [] New property found on endpoint response.
. [] Defined property missing from endpoint response.
. [] HTTP status code mismatch, or status code is not defined.

Note: The above is crossed out because chai-openapi-response-validator ended up satisfying the need and providing suitable errors without as much coding on our part.

Dependencies

Assumes the following work is complete:

Out of scope

This work does not include implementing the tests within Mocha. That will be covered in a separate ticket.

Event Timeline

The current title of this task is "Create schema validator method". Suggest changing that to (something like) "Create response body validator method".

Reason: if I'm understanding this task correctly, the point is to validate that response bodies from the actual endpoints match their spec. We already have a test to validate the schemas themselves (see here). We will probably need to update that test a bit as we go along, but that would be under a different task. For the purpose of this task, we're validate endpoint responses, not specs or schemas.

I'm also a little unclear on this checkbox: "Implement a reusable method that will accept two versions of the responses and confirm whether they match or not."

I'm understanding the task as creating a method (or whatever Javascript machinery we decide is appropriate) that accepts an endpoint response and an associated spec, then performs validation of the endpoint response against the spec. I don't understand how multiple versions of the responses come into play. Which makes me wonder if I'm missing the point.

BPirkle triaged this task as Medium priority.Oct 10 2024, 3:19 PM

Change #1080767 had a related patch set uploaded (by BPirkle; author: BPirkle):

[mediawiki/core@master] REST: ability to validate responses against response schemas in tests

https://gerrit.wikimedia.org/r/1080767

Some possible validator libraries:

Edit: dunno why I listed chai-json-schema twice. Removed the redundant entry.

On the chai-json-schema validator, I tried this out, and got far enough to use this to validate endpoint responses against the endpoint schema. The main difficulty I encountered was that OpenAPI expresses whether a value can be null via a "nullable" property, while pure json schema 4 expresses this via including "null" in the array of allowed types. If this turned out to be the only barrier, it could be solved by detecting "nullable" and adding "null" to the type array at test time. A not-ready-for-production experiment is patchset 5 of this gerrit change. One known limitation of this experiment is that it would fail if type is already an array.

None of that is really what we want - ideally we'd validate against an actual unmodified OpenAPI spec. I'm planning to try Chai OpenAPI Response Validator next, which will hopefully do that.

Initial experimentation looks good for chai-openapi-response-validator. Per its README, that package "Supports responses from axios, request-promise, supertest, superagent, and chai-http". And fortunately, the REST client used in API testing uses supertest internally.

I was able to toss together a rough working test without too much trouble. I'll try to refine this and see if run into any issues.

Change #1080767 merged by jenkins-bot:

[mediawiki/core@master] REST: ability to validate responses against response schemas in tests

https://gerrit.wikimedia.org/r/1080767