Page MenuHomePhabricator

RFC: Gradually move closer towards the `fetch` standard for RESTBase's internal request / response interfaces
Open, MediumPublic

Description

Browsers now implement high-level request abstractions around fetch, which are very similar to what we have been doing in RESTBase, but add some desirable features like

  • streaming requests and responses (incl. 'tee' support),
  • optional decoding via text(), json(), blob() accessors.
  • 'proper' Map for headers,
  • enforcement of cloning, and
  • support for redirects.

These APIs are going to be fairly ubiquitous, so I think we should keep an eye on them & evaluate whether we should gradually move towards a fetch like interface in preq and RESTBase.

Request object

The fetch request interface differs from preq primarily in

  • the positional URL parameter (optional in preq),
  • Headers in addition to Object (as with Response), and
  • the handling of the body parameter.

Handling POST / GET with query is a bit ugly in plain fetch:

var url = new URL("https://geo.example.org/api"),
    params = {lat:35.696233, long:139.570431}
Object.keys(params).forEach(key => url.searchParams.append(key, params[key]))
fetch(url).then(/* … */)

Response object

Instead of the plain Object, return a Response. Major differences are:

  • json, text, blob methods for the body, and access to a stream.
  • ok & other extra attributes.
  • Headers is a Map, not Object.
  • The Response needs to be clone()d before reusing it.
  • The query attribute is replaced with url.searchParams.

WhatWG streams

Streaming responses let us reduce memory consumption and time to first byte. It is basically a precondition for client-side content composition performance on slow connections, and is now supported in Chrome Canary 50.

Fetch responses are WhatWG streams by default, which are an incompatible evolution from Node's built-in streams. It's still early days, and the spec continues to evolve. There is a node-compatible and well-tested reference implementation written in ES6. The latest version of this does not seem to be available as a node module, but it doesn't look too hard to create one. The main functionality exposed in clients is currently ReadableStream, so we could consider only exposing this interface for now.

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)
GWicke added projects: Services, RESTBase.
GWicke subscribed.
GWicke updated the task description. (Show Details)
GWicke edited subscribers, added: mobrovac, Pchelolo, Eevans; removed: Aklapper.
GWicke renamed this task from Keep an eye on the `fetch` standard for RESTBase's internal request / response interfaces to RFC: Gradually move closer towards the `fetch` standard for RESTBase's internal request / response interfaces.Nov 9 2015, 5:19 PM

With https://github.com/wikimedia/restbase/pull/527 and related work we now have a need for a cleaner representation of compressed or streamed response objects. The fetch spec's text, json and blob methods look like a good fit for this, and would be a first step towards fetch-ification.

A challenge is that current RESTBase code expects body to be a decoded object or string. While it is fairly easy to add top-level text, json and blob accessors returning a promise, we will need to find a sane solution for handling this in templated handlers.

Options:

  • Represent text, json and blob as attributes in a templated-handler context, and implicitly resolve them by statically analyzing the template at compile time (Todo: check if this is really possible). Switch body to be a reference to the original body, in un-decoded form.

Strawman:

- somerequest:
    request:
      uri: ...
    response:
      status: 200
      body: {{ somerequest.json() }}
- finalize:
    return:
      status: 200
      body:
        hello: world
        something: somerequest.body

https://github.com/gwicke/node-fetch-polyfill has a fairly complete implementation of fetch with WhatWG streams. This is also used in https://github.com/wikimedia/node-serviceworker & https://github.com/wikimedia/node-serviceworker-proxy.

Currently, the adaptation of the whatwg stream to node streams is performed explicitly in the proxy module. For more general support, we would want to integrate this directly in hyperswitch's response handling. We would also need to refactor existing hyperswitch uses to no longer rely on the body being a string or buffer. The response would also become a Response object from node-fetch-polyfill.

GWicke triaged this task as Medium priority.Oct 12 2016, 9:26 PM
GWicke edited projects, added Services (later); removed Services.

I'm moving this to icebox. The ticket contains some interesting ideas and info and we definitely should consider it if we do a major refactor of how we do node.js, but I don't think we will work on this any time soon.