Adopt JSON as blubber's internal configuration format
Closed, ResolvedPublic

Description

While working on Blubberoid and its OpenAPI specification, the benefits of using JSON as the primary serialized config format are starting to stack up:

  1. OpenAPI specifications can provide descriptions of request and response objects for a given endpoint and accepted media type, but object schemas for a YAML media type are not supported; this is because:
  2. There is no media type for YAML registered with IANA. The closest I found to even a semi-official type is a proposal on the IETF mailing list which lacks a follow-up.
  3. OpenAPI specifications use JSON Schema as a means of defining acceptable data objects, but it does not work with YAML media types (again, there isn't a YAML media type). We could potentially have Blubberoid support JSON as well as YAML, but leaving the internal YAML-based parser would necessitate either duplicating the field descriptions and validation patterns, or writing our own JSON Schema generator that works from the yaml and validate field tags.
  4. Switching to JSON Schema for our own validation and configuration policy has potential benefits as well, namely one less external dependency (validator) and extensible schema rules.

Note that we may be able to retain YAML support for human-written configurations as well as support YAML on CLI and Blubberoid interfaces by converting YAML to JSON before unmarshalling. However, it isn't yet clear whether this can be done in a sane and efficient manner.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2018, 7:39 PM
dduvall updated the task description. (Show Details)Oct 22 2018, 11:25 PM
LarsWirzenius added a subscriber: LarsWirzenius.

I'll take this on.

LarsWirzenius triaged this task as Normal priority.Oct 23 2018, 3:41 PM

Change 471277 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[blubber@master] [WIP] Update library to convert YAML to JSON

https://gerrit.wikimedia.org/r/471277

hashar added a subscriber: hashar.Nov 5 2018, 1:56 PM

As I understand it that is for blubber internals, can you confirm we would still have the repository configuration held with yaml (.pipeline/blubber.yaml)? YAML is, to me, friendlier (comments, optionally no brackets, curly braces etc) and has a few useful tricks (aliases).

4 Switching to JSON Schema for our own validation and configuration policy has potential benefits as well, namely one less external dependency (validator) and extensible schema rules.

<rant>Or we could use XML with DTD for a formal definition/validation and XSLT for nice formatting. But I am getting old :D </rant>

As I understand it that is for blubber internals, can you confirm we would still have the repository configuration held with yaml (.pipeline/blubber.yaml)? YAML is, to me, friendlier (comments, optionally no brackets, curly braces etc) and has a few useful tricks (aliases).

That's the current plan. Accept input as either JSON or YAML (since JSON is valid yaml) and convert internally to JSON.

hashar removed a subscriber: hashar.Nov 30 2018, 3:25 PM

Change 471277 merged by jenkins-bot:
[blubber@master] Use JSON as canonical config format

https://gerrit.wikimedia.org/r/471277

thcipriani closed this task as Resolved.