Page MenuHomePhabricator

Batch reading list operations
Closed, ResolvedPublic

Description

To support apps which want to collect many changes and submit them together (to preserve bandwidth, or improve performance, or deal with being offline), reading lists should support batch operations (e.g. creating or deleting many list items at once).

To decide:

  • which operations should this include? (list item creation/deletion? list creation/deletion/update?) *all of those*
  • should there be a batch version of each endpoint (e.g. POST /lists/{id}/entries/ could take an array of list objects instead of a single one - a bunch of work but would make it possible to implment them efficiently with batched DB queries) or one single batch that takes a list of [operation_type, param1, param2...] items (easy to do, but we'll have to translate them to lots of single-row DB writes)? *extra parameter for each entry point*
  • how do you pass an arbitrarily long list of paramters to the action API? *JSON in parameter*
    • use an x-www-form-urlencoded parameter such as project1|title1|project2|title2|... combined with unit separators (ugh)
    • use a JSON-encoded string as an x-www-form-urlencoded parameter (double ugh)
    • use JSON in the POST body (a departure from API conventions; we claim that the API supports XML etc. clients as well)

To support T184680: Update duplicate handling in reading lists API, batch responses (at least in the REST API) should be in the form of [{"id":1}, {"id":2}] not {"id":[1,2]}

Todo:

Event Timeline

Tgr created this task.Dec 5 2017, 1:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Per T182475, using an API parameter that contains all real parameters JSON-encoded into a string seems like the best approach, so probably add a batch parameter to the create, update, delete, createentry, deleteentry endpoints and update all the repository methods to accept arrays of ids / list fields. (For update that would mean changing the query into INSERT ... ON DUPLICATE KEY UPDATE, except that's probably unsafe with statement-based replication (do we still use that?). So maybe just do a bunch of single-row updates. Or maybe not bother with batching updates at all.) Clients will have to take care to send things in a safe order (e.g. list creations before list item creations).

Tgr claimed this task.Jan 9 2018, 5:56 PM

Change 403361 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/ReadingLists@master] Support batch operations

https://gerrit.wikimedia.org/r/403361

Change 403361 merged by jenkins-bot:
[mediawiki/extensions/ReadingLists@master] Support batch operations

https://gerrit.wikimedia.org/r/403361

Tgr updated the task description. (Show Details)Jan 11 2018, 6:18 AM
Tgr updated the task description. (Show Details)Jan 11 2018, 6:25 AM
Tgr added a comment.EditedJan 15 2018, 12:29 PM

The associated RESTBase patch is #944.

Tgr updated the task description. (Show Details)Jan 15 2018, 11:35 PM
Tgr updated the task description. (Show Details)
Tgr updated the task description. (Show Details)Feb 12 2018, 11:13 PM
Tgr closed this task as Resolved.Feb 12 2018, 11:16 PM
Tgr updated the task description. (Show Details)

Broke out the non-central parts of the task into separate tickets.

@Tgr is batch deletion of reading lists and entries implemented? I don't see it in the documentation, but is it just not documented or not available yet? Thanks

Tgr added a comment.Feb 16 2018, 8:07 PM

It was at some point but I removed it per feedback on the patch.