There's multiple levels of reasoning why this will probably not happen:

1. Fetching by page_id is hard. Summaries as all the rest of the content in RESTBase are indexed by title, and secondary indexing them by page_id is quite some work (performance-vise, not implementation work). Also, fetching something by a secondary index is effectively queuing 2 tables inn Cassandra instead of one, thus almost duplicating the latency. Last but not least - with the new storage design we're working on we might get rid of secondary indexes all together.

2. Supporting bulk access with effectively eliminate Varnish cashing for those requests. In RESTBase each summary will still be queried individually, so the overall latency for the client will probably increase compared to making individual requests.

3. With http/2 the connections are shared, so making a lot of small requests shouldn't be an issue.

We can try to prototype this and measure the performance, but I doubt the results would be appealing.

[...] it would be a huge benefit if clients could pass the page ids (or titles) returned by these APIs and then receive the summaries as an array (potentially paging large result sets)

I may be misinterpreting the ticket, but it seems to me that the only huge benefit here is that clients wouldn't need to loop through the result set obtained from MW. In general, I understand your argument. Ideally, it would be possible to have the REST API return results based on an arbitrary (but yet unique) field. Unfortunately, the reality of the situation is that it comes at a great cost (on all levels)...

In T164083#3221334, @Pchelolo wrote:

We can try to prototype this and measure the performance, but I doubt the results would be appealing.

I don't see how this would fit in the new schema design plan either, so I wouldn't be keen on trying this, TBH.

Sorry let me descope this a bit here:

Lets focus on the list rather than page ids… (that is a secondary concern) Sorry to conflate the issues

@mobrovac @Pchelolo so now this is just about returning the list…

I may be misinterpreting the ticket, but it seems to me that the only huge benefit here is that clients wouldn't need to loop through the result set obtained from MW.

Yes, thats the main idea here. Its much better for clients (especially mobile clients) to get 25 results from MW and then be able to get all those 25 summaries from RESTBase in one call.

In this case the client would only need to make 2 calls, rather than the current situation: n+1, or 26 calls.

Supporting bulk access with effectively eliminate Varnish cashing for those requests. In RESTBase each summary will still be queried individually, so the overall latency for the client will probably increase compared to making individual requests.

Thats what I was afraid of. This is going to make fetching summaries for Reading Lists a bit of a bear for clients. it's possible, its just a lot of requests to different end points. But if thats the way it has to be, thats the way it has to be.

In T164083#3221334, @Pchelolo wrote:

3. With http/2 the connections are shared, so making a lot of small requests shouldn't be an issue.

On the other hand having to piece together the list from requests to a dozen different domains does have a cost. At a mininum extra DNS lookups will have to be done, and connection sharing between multiple domains seems to be problematic as well.

Clearly we don't want Varnish to store a copy of the same article for every domain, so I wonder if it's possible to do some creative rewriting? Say, set up something like wikimedia.org/api/shared/en.wikipedia.org/rest_v1 and then have Varnish rewrite it to en.wikipedia.org/api/rest_v1 before doing the cache lookup.

Reading lists will be user-centric, so perhaps the service can assemble those on the back-end?

In T164083#3229626, @mobrovac wrote:

Reading lists will be user-centric, so perhaps the service can assemble those on the back-end?

+1. This also avoids the DNS latency issues. Server side caching should not be a major concern for per-user reading lists.