Page MenuHomePhabricator

[Discussion] Consider validating JSON schemas when running x-ample tests?
Open, MediumPublic

Description

Swagger spec entry points potentially specify a JSON spec for the return value. Currently, we ignore this spec in x-ample tests, which leads to some amount of duplication. A lot of x-ample response assertions are only asserting the presence of mandatory attributes, which is also covered by the schema.

I think it would be attractive to validate responses against the JSON schema when available, before executing other x-ample assertions. This way we can reduce the duplication between x-ample and the schema, and make sure that the schema reflects actual output.

@mobrovac, @Eevans, @bearND, @Yurik, @Joe, @Pchelolo, @Sniedzielski, @Arlolra: Any thoughts?

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)

@GWicke, this is great! My background for this ask is:

  1. I would love to be able to write specs like the Swagger pet store[0] (and other) examples, which I assume to be canonical, and end up with something that looks standard but still keeps all the great test automation!
  2. The ability to specify models seems of great value for many reasons. A couple are the ability to specify optional and typed properties, which I don't think x-amples supports.

[0] https://github.com/swagger-api/swagger-spec/blob/master/examples/v2.0/yaml/petstore-expanded.yaml

In general, I'm much in favour of de-duplicating effort. In fact, ideally the services' code bases would be Swagger-driven, so that when you add a route, you don't need to remember to add it somewhere else as well.

In this particular instance, though, that would mean modifying the nagios checker script to parse the entire spec and implement most Swagger semantics, so I'd welcome @Joe's opinion on this one before taking any action.

mobrovac set Security to None.