Page MenuHomePhabricator

[RFC] Generalize POST parameter to JSON structure and header mapping in REST APIs
Closed, DeclinedPublic

Description

Most of our POST end points currently accept multipart/form-data / application/x-www-form-urlencoded or JSON. However, only JSON can represent nested structures, which is occasionally useful to avoid very repetitive names.

Idea: Reuse dot notation from TOML

Tom's Obvious, Minimal Language (TOML) maps nested object structures to dotted path syntax, and creates implied objects on demand. We could copy this idea to represent nested structures in POST keys.

Examples:

JSON:

{ 
  "foo" : { 
    "bar": "one",
    "baz": "two"
  },
  "quux", "boooz"
}

POST data:

foo.bar: one
foo.baz: two
quux: booz

Using this scheme to support overriding HTTP method or headers

We could consider leveraging this scheme to better and generally support HTML form interaction with REST APIs. HTML forms still don't support methods other than GET or POST (without JavaScript) or the specification of request headers, which complicates the creation of form front-ends for REST APIs. Some REST APIs offer ad-hoc header-based work-arounds for the method issue, but there doesn't seem to be anything general. As a result, APIs that need to be usable from HTML forms come up with custom semantics / POST keys, and generally can't use the full set of HTTP methods and headers. To avoid this, we could consider defining a general _http. override object like this:

_http.method: put
_http.headers.if-match: "679786764/581593fe-54d0-11e5-96a3-1b0c6675c2e8"

In the service processing the POST request (RESTBase, for example), this could be pulled out of the POST body in a general pre-processing pass on a POST request, before any routing or security checking has taken place. To avoid abuse, we could restrict this to a small white-list of methods and headers.

See also T101501#1614873 for the original discussion of this issue.

Event Timeline

GWicke raised the priority of this task from to Medium.
GWicke updated the task description. (Show Details)

+1 as long as we don't mix JSON and TOML and allow users to specify the same thing in JSON as well.

A consideration that came up in the context of T116336 is that generally moving to JSON request bodies would eliminate Swagger-UI form fields for current POST parameters. Those are very useful for testing, so it would be bad to lose this functionality entirely.

There is a JSON editor in latest swagger-ui, but I haven't seen a demo of that yet, so not sure how that compares to the current form.

Edit: @Pchelolo tried the editor, and it is not currently as usable as the forms.

Some further considerations:

  • Request format versioning: JSON content types can be versioned using standard content-type headers. Form data would have to be versioned using an in-band mechanism, using a post data parameter, perhaps along the lines of the pagebundle structure. The need to change request formats in backwards-incompatible ways is generally not as common as for responses, but it is pretty clear that we are going to need it to avoid incrementing major API versions on each breaking request format change to a single stable API end point.
  • Client support: Some clients have better support for form posts than raw JSON posts, while others have better support for pure JSON.

An intermediate option we could consider is:

  • Default to forms for the top-level properties, but still support pure-JSON POSTs.
  • Type JSON parameters throughout. For JSON parameters, use a type of object with a schema.
  • Coerce form posts, leave JSON POSTs untouched.
  • Validate the entire POST body (after coercion) against a schema derived from form definition & property types (and possibly schemas).

Pros:

  • Preserves the form interface in the docs.
  • Maintains support for pure JSON posts.
  • Compatible with both JSON & form centric clients.
  • Supports efficient POSTing of binary data via multi-part form encoding.

Cons:

  • Limited to inline / less consistent request format versioning.

@Pchelolo and I looked into implementing the form-primary option described above yesterday, but then found that the OAI / Swagger spec does not actually support specifying schemas (or even a type of object) for formData parameters. This means that right now, we have to choose between

a) Pure-JSON POSTs with full request body schema support, or
b) Form data posts without schema support for object parameters (just string).

We could lobby for supporting an object type with a schema for formData parameters in a future version of the spec. Until that is supported, we could document those parameters as strings, with example JSON values in the description (markdown supported).

@mobrovac, @Eevans, @Pchelolo: Does this sound like a good way forward to you?

It's not clear to me which way forward are you promoting @GWicke. To me it seems like going with (a) is cleaner; expecting clients to support sending application/json content does not seem like imposing too hard a requirement on them.

@mobrovac: The issue with pure-JSON body parameters is that the documentation sucks, and not all clients support it. With current Swagger-UI, we'd lose the form fields in the docs. There is some JS JSON editor available that we could enable, but it is pretty hard to use compared to the current forms.

In principle, the UI issues are separate from the spec itself. There is nothing stopping us from extending Swagger-UI to show a form interface for the top-level properties in a JSON spec.

In either solution, both JSON POSTs and plain form posts are supported. The issue is mostly about the docs.

I'm still leaning towards the "intermediate" solution described in T111748#2603225, but could live with a JSON-primary + UI changes solution as well. It looks like a bunch of work to support that well in the docs, though.

@Pchelolo implemented the JSON schema centric solution at https://github.com/wikimedia/hyperswitch/pull/59. While the current doc UI for JSON bodies are not that awesome, this solution does lead to somewhat cleaner code, and has the big advantage of supporting schema validation for the full request. Form posts are still supported, but are not documented prominently for now.

We basically have to make a choice between doc usability and schema support, and @Pchelolo's solution goes with schema support. Long term, we can work with the OAI folks to improve the doc experience for JSON requests in swagger-ui.

@Pchelolo implemented the JSON schema centric solution at https://github.com/wikimedia/hyperswitch/pull/59. While the current doc UI for JSON bodies are not that awesome, this solution does lead to somewhat cleaner code, and has the big advantage of supporting schema validation for the full request. Form posts are still supported, but are not documented prominently for now.

We basically have to make a choice between doc usability and schema support, and @Pchelolo's solution goes with schema support. Long term, we can work with the OAI folks to improve the doc experience for JSON requests in swagger-ui.

I'd second this route - validation and schema support trump docs and presentation in my books.

I don't think we need this, nothing is wrong with postinng application/json