Page MenuHomePhabricator

Enable RESTBase caching and storage for MobileApps end points
Closed, ResolvedPublic

Description

The MobileApps service is about to be ready for initial deployment. To that end, we need to hook it up in RESTBase :

  • create the Swagger spec
  • add the back-end service config

Several app service entry points will request the latest data for a title, without passing in a revision ID. This means that we'll need to check for the latest revision in the RB revision table before requesting mobile app content from storage and the service.

Pre-generation of mobile content on edit is not necessarily needed for the first deploy, but will be very desirable for performance later on. We are investigating solutions for change propagation this quarter (T102476), which should provide a solid mechanism for app service updates as well.

The current plan is also to store lead and remainder sections, both wrapped in JSON (see T102270). It might be desirable for the app service to return both in one JSON blob. The app will generally request the parts separately, but there is also the possibility of requesting both parts at once. In RESTBase, we can implement this by storing the service response as a single JSON blob. For lead / remainder requests, we can simply return the lead/remainder sub-objects, which are already parsed in RESTBase.

Zero header support in Varnish (T102524) would be desirable, but likely won't be a hard blocker for the service roll-out. The main effect of this on the app is showing a 'you are browsing for free' banner, which might be okay to be missing in the beta channel.

Event Timeline

mobrovac claimed this task.
mobrovac raised the priority of this task from to Medium.
mobrovac updated the task description. (Show Details)
mobrovac added subscribers: mobrovac, bearND.
mobrovac raised the priority of this task from Medium to High.Jun 18 2015, 6:21 PM
bearND set Security to None.
bearND moved this task from Incoming to Tracking on the Mobile-Content-Service board.

Please see https://phabricator.wikimedia.org/T103811#1459331 for a discussion of possible public end points.

Well, the whole ticket T103811 is relevant :P Also note that the discussion there is related to public endpoints as exposed by RESTBase.

PR #281 aims at adding the simple routes to RESTBase (only proxying for now).

mobrovac lowered the priority of this task from High to Medium.Jul 23 2015, 9:32 AM

I'm lowering the priority since the PR has been merged, but there are still TODOs to tackle here, which will need to wait.

(moving the thread back from T103811):

@bearND:

If the 'lite' version provides basically textual content, how about calling it 'mobile-text' ? The 'hybrid' version.. maybe mobile-html or mobile-simplehtml?

@GWicke I could go with mobile-text for the lite version.
For the hybrid one maybe something like mobile-sectioned-html or mobile-sect-html? I admit it sounds a bit weird but it basically provides the HTML in section JSON objects (+ extra metadata, of course).

mobile-sect-html

That made me smile ;) Clearly the mobile sect's HTML, or something like that.

Another option: mobile-html-sections, mobile-html-sections-lead and mobile-html-sections-remaining

@GWicke Now we're getting somewhere. I like your option:
mobile-html-sections, mobile-html-sections-lead and mobile-html-sections-remaining

@GWicke The Content-type would still be JSON, though. Would that be confusing?

@bearND, there is a lot of precedent for that, for example data-parsoid and all the listings.

A nice content type can even link to your format spec: application/json; profile=mediawiki.org/wiki/Wikimedia_Apps/API/sections/1.0.0

@GWicke Sounds great. So we'd have:

/{domain}/v1
-- /page
   -- /mobile-text/{title}                    # for lite app
   -- /mobile-html-sections/{title}           # for saved pages in regular app
   -- /mobile-html-sections-lead/{title}      # for enh. link preview and beginning of actual content in regular app
   -- /mobile-html-sections-remaining/{title} # for remainder of actual content in regular app

Are there some good examples for format specs? I've tried https://www.mediawiki.org/wiki/Specs/html/1.1.0 which I got from the content-type header of https://restbase.wikimedia.org:443/en.wikipedia.org/v1/page/html/Cat. But no such page exists.

@Dbrant and others from the mobile apps team. Does this look good? Note, some of this discussion was on T103811 as well.

Change 229419 had a related patch set uploaded (by BearND):
Change mobile/app/page/lite to page/mobile-text

https://gerrit.wikimedia.org/r/229419

Change 229421 had a related patch set uploaded (by BearND):
Add mobile-html-sections route

https://gerrit.wikimedia.org/r/229421

Change 229419 merged by Mobrovac:
Change mobile/app/page/lite to page/mobile-text

https://gerrit.wikimedia.org/r/229419

Change 229421 merged by Mobrovac:
Add mobile-html-sections route

https://gerrit.wikimedia.org/r/229421

PR #302 makes RESTBase expose and use the new routes.

PR #302 makes RESTBase expose and use the new routes.

The PR has been merged. Note that, as its predecessor, it only proxies the requests, no storage is involved. The next step is to include storing responses as well.

Because mobile-html-sections-lead and mobile-html-sections-remaining are only reusing the output of mobile-html-sections (and return only one of its components), RESTBase could query for and store only that response, and then serve requests for the other two by simply extracting the relative fields from the stored JSON.

Change 232003 had a related patch set uploaded (by Mobrovac):
RESTBase: Add MobileApps endpoints and back-end config

https://gerrit.wikimedia.org/r/232003

Change 232003 merged by Giuseppe Lavagetto:
RESTBase: Add MobileApps endpoints and back-end config

https://gerrit.wikimedia.org/r/232003

I suppose that's done ?

In the sense of blocking the deployment, yes, but it has not been completed yet. RESTBase is currently only proxying requests and forwarding them to the MobileApps service, but we want to expand this with storage/caching behaviour.

I'll remove the blocker, but keep this ticket open for reference.

GWicke renamed this task from Hook up the MobileApps service with RESTBase to Add caching & storage for MobileApps RESTBase end points.Oct 3 2015, 1:31 AM

Changed the title to reflect what's left.

mobrovac renamed this task from Add caching & storage for MobileApps RESTBase end points to Implement RESTBase caching and storage for MobileApps end points.Oct 4 2015, 9:59 AM
GWicke renamed this task from Implement RESTBase caching and storage for MobileApps end points to Enable RESTBase caching and storage for MobileApps end points.Oct 5 2015, 10:28 PM

Change 247935 had a related patch set uploaded (by Mobrovac):
RESTBase: Set up MobileApps storage and AQS public API

https://gerrit.wikimedia.org/r/247935

Change 248026 had a related patch set uploaded (by Mobrovac):
RESTBase: Set up MobileApps storage

https://gerrit.wikimedia.org/r/248026

Change 248026 merged by Alexandros Kosiaris:
RESTBase: Set up MobileApps storage

https://gerrit.wikimedia.org/r/248026

PR 420 sets up new storage and uses the new-style public routes.

Change 253895 had a related patch set uploaded (by Mobrovac):
RESTBase: Config: Move MobileApps back-end spec out of config

https://gerrit.wikimedia.org/r/253895

Change 253895 merged by Filippo Giunchedi:
RESTBase: Config: Move MobileApps back-end spec out of config

https://gerrit.wikimedia.org/r/253895

PR #471 enables the pre-generation of content for the Mobile Content Service.

This is basically done, so resolving. We are now looking into why Varnish doesn't seem to accept our purging requests in T127370: Varnish not purging RESTBase URIs.