Maniphest T191795

Support MCR in mobile
Open, Needs TriagePublic
Actions

Description

The service stacks that support mobile (web/Android/iOS) need to support multi-content revisions (see especially the use cases there for background). This is an exploratory task to figure out the shape of that.

In the current PHP stack, some storage service looks up the content and metadata for the current revision, the content handler generates a HTML blob from it (in the case of wikitext this involves the parser and extension callbacks), the skin takes the content HTML and the revision metadata and renders the final page. For mobile, that is replaced by client-side rendering: the app fetches the revision metadata (via the PHP API or RESTBase/MCS), it fetches the content blob (via Parsoid/RESTBase/MCS, or by calling the PHP parser via the MobileFrontend API), and renders the final page.

In the new post-MCR PHP stack there will be more steps: the storage service looks up the revision metadata, and the slot metadata and content for each revision slot, the content handlers generate a HTML blob from each slot content, these slots get combined into a single HTML blob, and that gets handed along with revision metadata to the skin. So there will be a step that's largely analogous to the old rendering process (slot content -> slot HTML) and a new "combining" step (multiple pieces of slot HTML -> single multi-content HTML). Alternative renderings that don't use MediaWiki skin (and possibly parsing) logic will need to reproduce this.

(Note that there is significant uncertainty at this point around the "combining" step - it might pretty much just concatenation, or it might be more complex. It might be some transformation starting with a set of fully rendered slot HTMLs, or it might be more integrated with the parsing process.)

Specifically, this probably means that

Parsoid needs to be able to address single slots (i.e. handle routes like /page/html/<title>/<slot> and /page/html/<title>/<revision>/<slot>) and handle invalidation correctly (slots can depend on other slots of the same revision)
something (probably MCS?) in the Node stack needs to be able to combine a bunch of slot HTMLs into a single HTML so that apps can show a single-page rendering of any content they don't have any special handling for. This is probably going to be fairly simple (just wrap them in some HTML container and concatenate).
in RESTBase, /page/html/<title> and /page/html/<title>/<revision> should maybe redirect to that combining service. (Or maybe it should only show the main slot - not clear which would be the more sensible behavior for old clients.)
apps that use core MediaWiki API calls will have to implement HTML combining on their side, or the APIs might take care of that themselves when no slot parameter is present. I don't think there is any major app still using such calls.
for apps that use MobileFrontend's mobileview API, that API needs to do the HTML combining (there will be a PHP service for that).

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	BUG REPORT	None	T12863 Links on commons upload summaries do not link to commons
Open		None	T5498 Image history is confusing
Stalled		None	T96384 Integrate file revisions with description page history
Open		None	T115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
Open		None	T115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
Duplicate		None	T115864 Figure out if the description of the image can be used as the caption on-wiki
Open	Feature	None	T53032 When inserting an image, set its caption by default to be the Commons image description
Open	Feature	None	T39534 Wikimedia Commons should support searching by color
Duplicate		None	T39535 Wikimedia Commons should support filtering by color
Resolved		None	T19503 Provide metadata support on Wikimedia Commons
Resolved		None	T122038 Moving/Deleting a video file does not move/delete subtitles files, nor does it inform the file mover of their existence
Open		None	T135221 Make TimedText content an integral part of the File page
Open	Feature	None	T2167 Use a dedicated interface for adding meta-data like interwiki links, rather than wikitext
Open		None	T132072 Integrate page meta-data as a new content model revision slot for consistency and atomicity
Resolved		None	T51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
Resolved		None	T68108 [Epic] Store media information for files on Wikimedia Commons as structured data
Open	Feature	None	T14963 allow per-page exceptions to spam blacklist
Open		None	T203157 Make the spam whitelist its own slot
Open	Feature	None	T56140 Move TemplateData to its own JSON-content namespace and associate with Template-namespace, or to its own TemplateData content model and revision slot
Stalled		None	T174043 Deploy Multi-Content Revisions
Open		None	T174022 Implement multi-content revisions
Open		None	T189220 Ensure that relevant extensions are MCR-aware
Open		None	T191795 Support MCR in mobile

Event Timeline

Tgr created this task.Apr 9 2018, 1:43 PM

Restricted Application added a project: Product-Infrastructure-Team-Backlog-Deprecated. · View Herald TranscriptApr 9 2018, 1:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Tgr updated the task description. (Show Details)Apr 9 2018, 1:45 PM

Tgr updated the task description. (Show Details)

• Pchelolo added a project: Services (designing).Apr 9 2018, 1:47 PM

Tgr updated the task description. (Show Details)Apr 9 2018, 1:49 PM

• Mholloway subscribed.Apr 9 2018, 1:50 PM

The other option that was discussed for how HTML generation works internally in MCR is that there will be a single HTML blob, which is initially the HTML rendering of the content of the main slot, and then handlers for the other slots somehow expand that progressively. That seems very hard to reproduce in the Node stack, though.

Parsoid needs to be able to address single slots (i.e. handle routes like /page/html/<title>/<slot> and /page/html/<title>/<revision>/<slot>)

Bikeshedding:
(1) s/page/pageslot
(2) Better routes may be /pageslot/<slotid>/html/<title>(/<revision>)? or /pageslot/html/<slot>/<title>(/<revision>)? .. otherwise, there is ambiguity in the revision and slot parsing in RESTBase.

Slot IDs will be short strings (like main or documentation), not numbers. Although currently we don't verify that it's not a numeric string; we probably should.

Tgr updated the task description. (Show Details)Apr 9 2018, 2:09 PM

In T191795, @Tgr wrote:

Alternative renderings that don't use MediaWiki skin (and possibly parsing) logic will need to reproduce this.

At least for the first versions of MCR, skins aren't planned on being changed. But anything that gets "the" wikitext of a page to parse it itself (e.g. via Parsoid) will need to be able to deal with combining multiple slots in MCR.

In T191795#4116745, @Tgr wrote:

The other option that was discussed for how HTML generation works internally in MCR is that there will be a single HTML blob, which is initially the HTML rendering of the content of the main slot, and then handlers for the other slots somehow expand that progressively. That seems very hard to reproduce in the Node stack, though.

That's not likely to be hard to reproduce in node, just annoying because of code duplication.

In T191795, @Tgr wrote:

the content handlers generate a HTML blob from each slot content, these slots get combined into a single HTML blob
[...]
and a new "combining" step (multiple pieces of slot HTML -> single multi-content HTML)

The details are what matter here.

Blindly concatenate the HTML blobs? Probably easy to duplicate in node.
Have some sort of per-page-type "template" into which the blobs get plugged in? You'd need to be able to get the right template into node. Which might be problematic if the "template" depends on the page metadata in some way.
Have the option to add different HTML than whatever blob the Content produces standalone?
Find markers of some sort in one slot's HTML to indicate where another's HTML is inserted?
Combinations of the above depending on the specific slots involved?
Something else?

In T191795, @Tgr wrote:

apps that use core MediaWiki API calls will have to implement HTML combining on their side (the APIs will support per-slot functionality). I don't think there is any major app still doing that.

Chances are that apps that use the action API action=parse and specify page, pageid, or oldid to get page content will just work, since the most likely implementation will be to return the combined HTML unless per-slot HTML is specifically requested (with a new parameter).

In T191795#4116792, @Tgr wrote:

Slot IDs will be short strings (like main or documentation), not numbers. Although currently we don't verify that it's not a numeric string; we probably should.

Why should we? There's no reason a slot role name couldn't be "12345" if that makes sense in context.

In T191795#4116802, @Anomie wrote:

Why should we? There's no reason a slot role name couldn't be "12345" if that makes sense in context.

I doubt there's a context in which that makes sense, and it seems like a useful guarantee for REST APIs. (Although such APIs might want to nevertheless avoid routes which only differ in what pattern the parameter matches, I suppose.)

something (probably MCS?) in the Node stack needs to be able to combine a bunch of slot HTMLs into a single HTML so that apps can show a single-page rendering of any content they don't have any special handling for. This is probably going to be fairly simple (just wrap them in some HTML container and concatenate).

In the general case, document composition will have to worry about https://www.mediawiki.org/wiki/User:SSastry_(WMF)/Notes/Document_Composability#Nesting_constraints ... But, in the simple case where the composition introduces the fragments as children of <body>, dealing with this will be the no-op constraint. Alternatively, we come up with a composition spec (see https://www.mediawiki.org/wiki/Parsing/Notes/HTML5#Composition_Spec_notes) that is applied uniformly in all html5 document composition scenarios everywhere in mediawiki.

Tgr updated the task description. (Show Details)Apr 9 2018, 2:20 PM

Tgr updated the task description. (Show Details)Apr 9 2018, 2:23 PM

• bearND subscribed.Apr 9 2018, 3:25 PM

• mobrovac added a project: User-mobrovac.Apr 11 2018, 7:02 PM

• Niedzielski added a project: Web-Team-Backlog-Archived.Apr 12 2018, 2:04 AM

• Niedzielski moved this task from Incoming to Tracking on the Web-Team-Backlog-Archived board.

• Niedzielski edited projects, added Web-Team-Backlog-Archived (Tracking); removed Web-Team-Backlog-Archived.

• Niedzielski added a subscriber: Jdlrobson.

• Niedzielski added a subscriber: pmiazga.

• Niedzielski subscribed.

daniel moved this task from Inbox to Epic on the Multi-Content-Revisions board.May 7 2018, 10:42 AM

daniel moved this task from Epic to Watching on the Multi-Content-Revisions board.May 7 2018, 10:45 AM

ssastry moved this task from Needs Triage to Non-Parsing-Team Tasks on the Parsoid board.Sep 20 2018, 5:15 PM

Tgr added a parent task: T189220: Ensure that relevant extensions are MCR-aware.Oct 3 2018, 1:05 AM

• mobrovac added a project: Platform Team Legacy (Designing).Dec 20 2018, 12:54 PM

• CCicalese_WMF added a project: Platform Engineering (MCR).Dec 20 2018, 1:45 PM

LGoto moved this task from Needs triage to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.Jan 2 2019, 4:50 PM

I just want to mention, that currently, the MobileFrontend allows fetching data from MCS instead of MediaWiki. Please refer to McsContentProvider class.
We're using it to verify how the HTML returned by /page/mobile-sections/{TITLE} looks on mobile. To test it locally please change the wgMFContentProviderClass to MobileFrontend\ContentProviders\McsContentProvider.
Also, the McsContentProvider allows to point it to different service, just change the wgMFMcsContentProviderBaseUri config variable (it's "https://en.wikipedia.org/api/rest_v1" by default)