Page MenuHomePhabricator

vNext: Decide on the schema for pagination envelopes
Open, MediumPublic5 Estimated Story Points

Description

REST endpoints that return a paginated list of items should use a common schema for the envelope that facilitates the pagination.

The general shape of the envelope should look like this:

{
  "items": [ ... ],
  "pagination": {
    "next": { ... }
  }
}

There should be a shared schema describing the envelope. Individual endpoints will then fill in the schema for elements of the "items" array using JSON schema's allOf mechanism. TBD: are items required to be objects for all endpoints?

The "next" entry in the "pagination" stanza is required, it will be null if there is no next page. We may add additional keys next to "next" in the future (required or optional is TBD): "first" would be easy, for endpoints that support inverse sorting we could also have "previous", "last", and "reverse". Implementing "previous" without the need to reverse the sort order is tricky - the pagination direction would dictate the database sort order, items may be re-sorted for output.

The biggest open question is the exact shape of the cursor used by "next" and friends. Options:

  • a JSON object of query parameters (to be merged with other parameters by the client). This is what the Action API already uses.
  • an opaque Base64 cursor, containing an encoded JSON object of query parameters (to be merged with other parameters by the server). This is the industry standard.
  • a full href pointing to the next page. This follows the linked-data paradigm but may not work well with how clients are typically implemented (with a configurable base URL).

Notes:

  • All three would map to an array of query parameters internally, and would be compatible with how the Action API currently implements pagination.
  • The options are not mutually exclusive: the envelope could carry more than one. Adding a mechanism later is easy. Removing one is not, so each extra field is a long-term commitment.
  • The same envelope is meant to be shared by native REST endpoints, Action-API-over-REST list routes and perhaps even a future GraphQL endpoint.
  • Cursors are not stable for unstable rankings (e.g. offset-based generator=search) and can drift or repeat across pages.
  • Pagination has to be based on a unique key; if that key is based on a mutable property, items may be skipped or duplicated: if Zack is renamed to Aaron while the client is looking at the page from Francis to Mandy, they will never see the entry for Zack or Aaron.

Event Timeline

My thoughts on pros and cons of the pagination shape:

A JSON object of query parameters to override client-side.

Pros:

  • What the Action API does today, so existing clients already know the pattern and the Action API continue block maps onto it directly.
  • Simple mental model: everything is visible, no magic, and clients get what they ask for.
  • Convenient for clients: they merge the object into their own request parameters.

Cons:

  • The client has to do the merging.
  • Tempting for clients to manipulate or fabricate values. Not immediately problematic but brittle (clients bind to structures that are not guaranteed).
  • Parameters not involved in pagination (filters) should not be repeated in the response, so the client has to remember the original (or at least, the previous) set of query parameters, to preserve the filter conditions.

Note:

  • Filter and position parameters are not distinguished (e.g. from=20206-08-15 is conceptually a filter, but pagination overrides the same underlying condition). The Action API has no such separation.
  • When paging over a filtered result, params can include a hash of filter params that are expected to stay constant, to avoid drift. May help to spot bugs.

Base64-encoded JSON object holding query params, merged by the server.

Pros:

  • The industry standard. Clients treat the cursor as an opaque token and pass it back.
  • Opaque by contract, so the encoded content can change without breaking clients.
  • The server controls merging semantics.
  • Different from the established Action API mechanism, clients need to change code.

Cons:

  • Clients send the cursor in addition to other (filter) parameters. Parameters not involved in pagination (filters) should not be repeated in the response, so the client has to remember the original (or at least, the previous) set of query parameters, to preserve the filter conditions.
  • Requires conceptual separation of filter parameters and position parameters, so that the cursor's position wins over e.g. a from filter when the database query is built.
  • The server overriding parameters with values hidden in the cursor can be surprising and confusing.

Notes:

  • From the client's perspective, this is still "merge pagination params into original params", the only difference being that there is just one parameter to set.

A full, host-relative link to the next page.

Pros:

  • Good for blind processing: a generic linked-data client just follows the link, with no knowledge of parameters.
  • The server decides how the next page is addressed, so the pagination internals can change without client changes.

Cons:

  • Awkward for clients that build URLs with a request builder from a configurable base URL. They would need a "raw" mode, which may be disruptive.
  • Hostile for clients that need to inject or change parameters (e.g. a different limit), because they have to parse and rewrite the URL.

Note:

  • Most be host-relative to avoid issues with internal vs extermal host names when processing internal API requests.

Initial opinion without putting much thought into it yet: I dislike the "full href" solution, but don't yet have a strong opinion between the other two "JSON object" vs "opaque cursor". I lean towards "opaque cursor", but I admittedly haven't carefully compared those two options.

My two cents: moving to the opaque cursor gets us industry-standard without moving too far away from the current model, and it lets us change the contents whenever we need to, which is enough to make this the top contender.

KineticPelagic set the point value for this task to 5.

My personal preference is the JSON object, the way the action API does it. But the opaque cursor is a close second. I don't like the href option, because it doesn't jive with how API clients are typically written.

My points against the opaque cursor: it's worse for logging and also for caching (though we probably won't cache paginated output anyway). And it interacts with "normal" query parameters in a "magic" way (some parameters are overwritten by the cursor, some are not). Also, it's big and ugly.