Next Generation Content Loading and Routing, in Practice
Closed, ResolvedPublic

"Like" token, awarded by Addshore."Mountain of Wealth" token, awarded by Jhernandez."Like" token, awarded by Jdforrester-WMF."Doubloon" token, awarded by Mholloway."Like" token, awarded by cscott."Like" token, awarded by bearND."Like" token, awarded by BGerstle-WMF."Love" token, awarded by Niedzielski.
Assigned To
Authored By
dr0ptp4kt, Oct 2 2015




Feedback at the summit suggested that server-side compositing was preferred over a SPA, as server-side compositing is less risky. Furthermore, lower hanging fruit were recommended to be plucked. T113066: [GOAL] Make Wikipedia more accessible to 2G connections has been updated accordingly, which is a Q3 FY2015-2016 Reading Web goal to take action to get at those lower hanging fruit using conventional MediaWiki PHP and JavaScript.


Definition of the problem: Can we speed up both the end user and developer experience?

Today if you're on a slow connection chances are you'll have problems accessing large Wikimedia pages.

And if you're a developer looking to rapidly build an API driven application or new feature, once you figure out how to call the APIs you'll probably need to do a lot of extra parsing and mashup work at the client, and then you'll need to figure out how to unburden your client from network contention.

Can we do better, much better for users on slow connections?

And can we unburden feature developers from API composition problems, while letting people who love middleware focus on these problems instead?

Background information and R&D

In WMF fiscal year 2015-2016 quarter 2 (October - December), Reading engineers will prototype a Parsoid-backed HTML5 web application that dramatically reduces first paint time for users on slow connections (e.g., 2G).

Expected outcome at the summit: Here's what we know. Proceed?

This is a two hour session to share learning from this R&D, discuss the viability of the approach for multiple form factors and client (and server!) platforms, and inform a number of summit proposals on the table.

More related tasks: Hey, that's my idea!

The likely approach looks like an API driven web frontend with two step loading (like the Android app) of RESTBase Node.js mediated Parsoid HTML and inter-server api.php calls and ServiceWorker route interception. Fallback for low/no-JavaScript might entail loading the first X bytes, with a user clickthrough to fetch an entire article with very low frills styling.

To keep it simple, the goal is to largely replicate the existing mobile web experience with this new generalizable architectural approach. If this approach is successful, in Q3 the plan would be to aggressively move this to the beta channel of the mobile web and initiate processes for a beta feature for the desktop web so that gadget, special page, and other interested maintainers can see how stuff works.

I know, I know

There are several known challenge areas to explore in the Q2 work, and we should discuss these further at the summit:

  • VE forward compatibility and pre-fetched Parsoid for rapid bootstrapping
  • Gadgets
  • Idiosyncratic Special: pages
  • ResourceLoader...resource loading
  • New MediaWiki API endpoints to support this approach
  • Packaging for third parties
  • No-frills null skin for third parties who can't do Node.js but who can benefit from basic mobile first, multiple form factor CSS.
  • Wikipedia Zero support (is already compatible, but needs to be considered)
  • Search engine findability
  • Analytics
  • CentralNotice

Current status

Follow these tasks to track the current R&D progress:

  • T111588 API-driven web front-end
  • T113066 Barack Obama article on English Wikipedia on 2G connection should fully load in under 15s (down from 50s)

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2015, 10:42 PM
brion added a subscriber: brion.Oct 2 2015, 10:48 PM
dr0ptp4kt set Security to None.Oct 2 2015, 10:52 PM
dr0ptp4kt added subscribers: Tnegrin, Krinkle, Catrope.
bd808 added a comment.Oct 2 2015, 11:03 PM

Some (hopefully) clarifying comments for @dr0ptp4kt's bullet points:

  • VE forward compatibility and pre-fetched Parsoid for rapid bootstrapping

i.e. can VE be setup to edit directly from the HTML delivered for the browser view and avoid a roundtrip to RESTBase/Parsoid to get specially annotated HTML.

  • Gadgets

i.e. if we are changing the HTML seen by the browser what guidance and assistance can we give to Gadget authors to update their DOM manipulation code to work with this HTML.

  • Idiosyncratic Special: pages

i.e. We have special pages that don't have corresponding Action API calls so a purely API driven frontend client would need a way to deal with them (this may be where a "null skin" comes in to get MW generated HTML without "site chrome").

  • New MediaWiki API endpoints to support this approach

RESTBase, Parsoid and any new wrapper services will rely on MediaWiki to provide access to wiki content and business logic. We will probably need to add new Action API actions to support doing things that today only have an interface via a special page or that we want to be able to do discreetly from a new front end client (e.g. interact with session/user state).

  • Packaging for third parties

i.e. If we have yet another service that becomes important for a Wikipedia-like MediaWiki experience we need to figure out how to provide that same experience to 3rd party MediaWiki users.

  • No-frills null skin for third parties who can't do Node.js but who can benefit from basic mobile first, multiple form factor CSS.

i.e. Wikimedia can support a multi-tiered application composed of many separate services if we get good performance and scalability improvements from it, but we shouldn't assume that this becomes table stakes for anyone wishing to use MediaWiki.

dr0ptp4kt removed a subscriber: Ettorre.Oct 2 2015, 11:10 PM
dr0ptp4kt updated the task description. (Show Details)Oct 2 2015, 11:14 PM
dr0ptp4kt added a subscriber: DFoy.
  • Idiosyncratic Special: pages

i.e. We have special pages that don't have corresponding Action API calls so a purely API driven frontend client would need a way to deal with them (this may be where a "null skin" comes in to get MW generated HTML without "site chrome").

As @bd808 is hinting at, I think the big question that needs answering is: what about all the functionality?

Meaning: reading is easy. But what about editing? Talk pages? History? User preferences? Notifications? Special:MovePage? Other special pages? Etc etc.

I think a key part of this process should be a systematic survey of mediawiki-core, to separate out the *non-reading* functionality. Then we need to classify it: is there an API for this, so that clients could reimplement? That is, is there an high-level API which gives you the same information as, say, the history page, but without formatting it into HTML, so that a client could implement a different visualization? Is there an API for [[Special:MovePage]]?[1] If there's not an API, is there a way to get minimal HTML (ie, a "no skin" version without unnecessary styling) that would be suitable for restyling and embedding in a web view?

At the end we'd have a roadmap:

  • A set of features, like reading, which have been fully refactored to support multiple clients/skins/UI paradigms
  • A set of features which we plan to refactor to expose high-level APIs for clients to use (say, [[Special:MovePage]])
  • A set of "long tail" features which most clients won't find worth re-skinning or implementing via an API, but which have "no skin" variants appropriate for easily embedding in a web view.
  • A set of "terrible no good" features which we've made a conscious decision to deprecate. (Hopefully a smallish list.)

This would also serve to document the plan for all the features which are not currently supported by the mobile client, to answer the charge that mobile is cherry-picking functionality and creating third-class users.

[1] I think the answers to many of these will be "yes, there's a way you could do that from a nonstandard client, but...". For example, I'm pretty sure [[Special:MovePage]] just outputs a form, and by forging the correct POST request (with some hand-waving about authentication), you could perform the move. But this isn't currently documented, and I don't think anyone feels that this is a stable API. That is, the developer contract is that if you change the names of the fields in the form on [[Special:MovePage]], you just have to change the PHP code to match --- no one feels like you have to notify API clients of the change, etc. So part of the survey would be to document all the different tasks which the standard desktop UI exposes, and the API required to trigger this from a nonstandard client, and then to make these APIs stable.

@bd808, @cscott, thanks!

@cscott, I think a practical way to handle documentation in Q2, and perhaps in the session at the summit, is on this task as a checklist (self-documenting) or as a (how apropos!) page. What do you think is the simplest / best way to capture this while we race through Q2 to the summit? CC @Jdlrobson, @phuedx, @Jhernandez.

In WMF fiscal year 2015-2016 quarter 2 (October - December), Reading engineers will prototype an HTML5 web application that dramatically reduces first paint time for users on slow connections (e.g., 2G).

This needs clarifying.
I would say that the real win here is reducing time to interactive/initial load time. First paint is only really improved on image heavy articles. The problem here is that users cannot even load articles on 2G.

@JKatzWMF and I were talking about this today and I wanted to capture it. What if we acknowledged that there were actually two applications here -- an editor stack that needed to support 100,000 users and an consumption app that needed to support millions.

I suspect we'd need to follow @cscott's plan for pulling out all the functionality in mediawiki, so I'm not sure what this buys us in practice.

GWicke added a comment.EditedOct 3 2015, 7:39 PM

To me, reinventing everything *right now* is not very interesting or realistic. I firmly believe that the only realistic path towards a more flexible and performant front-end experience is progressive enhancement. We need to be able to focus on a selection of important use cases first, while continuing to leverage existing server-side implementations in less critical features like rarely-used special pages or forms. This will require the definition of public metadata interfaces for page components (T105845), as well as improved API support for skin-free retrieval of page components including special pages (improved action=render, "null skin", see T114596).

We have already made good progress with API support for apps and client-side skins (especially for core editing and reading), but there are also still gaps. The experiment in the coming quarter should let us identify and possibly address some of those remaining gaps. We will also verify if we can cleanly integrate with existing server-side functionality, which will establish if progressive enhancement is indeed a viable option going forward.

Qgil added a subscriber: Qgil.Oct 3 2015, 8:56 PM

Congratulations! This is one of the 52 proposals that made it through the first deadline of the Wikimedia-Developer-Summit-2016 selection process. Please pay attention to the next one: > By 6 Nov 2015, all Summit proposals must have active discussions and a Summit plan documented in the description. Proposals not reaching this critical mass can continue at their own path out of the Summit.

@Qgil, thank you for the update. For the plan documentation, is there a particular format you'd recommend? Or is it somewhat open ended? I should ask: is the current level of detail actually adequate for the plan? If not, what should be expanded upon?

dr0ptp4kt updated the task description. (Show Details)Oct 6 2015, 10:28 PM

@Qgil, I have noted this as a 2 hour session in the description. CC @Jdlrobson, @phuedx, @Jhernandez.

I ask that this is scheduled early in the program so it can inform the other discussions.

I have also linked to a webpage where @Jdlrobson provided a written description of the Q2 work to augment the Phabricator tasks.

@dr0ptp4kt Sounds good. We'll flesh out session structure and topics through this month as we advance.

Jhernandez awarded a token.
Qgil added a comment.Oct 11 2015, 8:15 PM

The plan looks good, and I guess we can accommodate for the 2h. One comment just to be sure, I hope you are planning to start the discussion asap instead of starting it at the Summit from scratch. We want to make the most of our time at the event, just as much as the time before the event.

Qgil moved this task from Backlog to On track on the Wikimedia-Developer-Summit-2016 board.EditedOct 12 2015, 8:57 PM

Please assign this task to the person driving this proposal.

Also, could you edit the description and add the expected fields defined at , please?

Tnegrin assigned this task to dr0ptp4kt.Oct 13 2015, 5:12 AM
dr0ptp4kt updated the task description. (Show Details)Oct 13 2015, 9:09 PM

@Qgil, the Description has been updated to more directly use the verbiage:

"definition of the problem"
"expected outcome at the Summit"
"current status of the discussion
"links to background information
"related tasks in Phabricator"

The RFC discussion on T111588, API-driven web front-end, is germane to this summit discussion, and we should place our emphasis on that RFC while the R&D is conducted. The R&D occurring this quarter should help to inform the go-forward approach.

@Tnegrin has assigned the task to me.

dr0ptp4kt updated the task description. (Show Details)Oct 13 2015, 9:14 PM

Thank you. Can you associate other projects related to this proposal in addition to Wikimedia-Developer-Summit-2016, please? This way the watchers of those projects will be aware of this task.

@Qgil, when I click the "Save Blocked By Tasks" button while trying to make this task a blocking task inside of T111588 and T111588 (i.e., after I have clicked the "Select" button), on both of those pages it just sits there with a tinted UI. Are you able to use your Phabricator super powers to make T114542 block each of T111588 and T111588?

I did manage to make it a task blocking 1723663 without getting stuck on the tinted screen.

dr0ptp4kt updated the task description. (Show Details)Oct 14 2015, 11:32 PM
dr0ptp4kt added a subscriber: kevinator.
Nuria added a subscriber: Nuria.Oct 15 2015, 2:29 PM

In WMF fiscal year 2015-2016 quarter 2 (October - December), Reading engineers will prototype a Parsoid-backed HTML5 web application that dramatically reduces first paint time for users on slow connections (e.g., 2G).

I agree with @GWicke in that this seems a lot of reinventing to solve a old problem . Namely, fast delivery of a website to a mobile phone in a poor connection. Progressive enhancement seems a much easier path. Optimizing the critical rendering path in mobile goes through having a fast paint no-js option., Php output can be flushed as soon as it is ready and later be "dressed" with javascript should the phone and connection support it. Architecturally this seems a much simpler option that can deliver value right away without the addition of a new backend.

In WMF fiscal year 2015-2016 quarter 2 (October - December), Reading engineers will prototype a Parsoid-backed HTML5 web application that dramatically reduces first paint time for users on slow connections (e.g., 2G).

I'm also a little confused by this sentence. We've already developed two API-backed reading-optimised applications in the mobile apps. What are we expecting to learn from developing a third one, that we didn't learn from developing the first two?

Nuria added a comment.Oct 15 2015, 3:45 PM

Old but (very) relevant. Why progressive enhancement is important:

DarTar added a subscriber: DarTar.Oct 16 2015, 4:03 AM

@Nuria Fast no-JS serving of HTML that will be progressively enhanced is what we're shooting for. Leveraging web db's, app cache and service worker to cache as much as possible, and the smaller/faster API calls for non-cached content.

Without a chrome/UI you can render in the server it becomes a really complicated problem to render on the server and hydrate the web app on the client seamlessly. The plan is to reuse the same views on the client and the server for rendering the chrome.

Another novel approach is the use of structured parsoid html for the data, which should give us a consistent view to the content in order to transform it as needed.

Also the use of restbase services should provide good enough API caching to handle the load of something as big as Mobile Web. We can't do that with the current api.php.

@Deskana ^ this is the approach. The focus architecturally is to push the mobile web to where the apps are going. The apps thick clients are moving (will be moving) to using properly cached restbase services, that will perform aggregation and massaging of the data. That is done now in the clients making them more inefficient and duplicating work per platform.

The web's plan is to use the lessons learned from their experience and use the same approach (where they are moving to), trying to move the mobile web platform forward in speed, UX, development experience.

In general, the approach to the experiment is to try the architecture that the native apps have (because there is no other way for them) and it provides of certain advantages (like client side caching and navigation, allowing for a snappier experience) but adapted to work sensibly with the web (progressive enhancement) and using the oportunity to serve a lightweight fast html-only content for mobile users from parsoid-html.

Of course as others have mentioned there is a ton of functionality that seems like we would have to rebuild/reinvent. As far as I know, the native apps have been asking questions about how to perform this functionalities, and there is no answer but linking to the browser's mw url. At least, on a website like the one mentioned on this experiment we are already on a browser and can link to the normal mediawiki links, and you're always able to do everything.

Here are some notes that informed the R&D project:

Comments and input greatly appreciated.

dr0ptp4kt moved this task from Backlog to Current Quarter on the Reading-Admin board.
dr0ptp4kt updated the task description. (Show Details)Oct 21 2015, 8:45 PM
dr0ptp4kt added subscribers: K4-713, atgo.
Tbayer added a subscriber: Tbayer.Nov 3 2015, 8:35 PM
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 3 2015, 8:35 PM

Note Tim's comment in T119022: WikiDev 16 working area: Content format

T114542 is a no-brainer, and T111588 and T106099 are closely related. It seems like they've been talking past each other a bit, so it would be good to have both Adam and Gabriel in the same room so we can get to the bottom of it. Adam is asking for 2 hours for T114542 -- I think 2-3 hours for these three RFCs combined may be reasonable, under the umbrella of T114542.

Given that this is an area that clearly needs discussion of some variety, should we indeed do it as Tim proposes? If so, what would the goal of the meeting be?

Note, I believe the longest schedule slot we are going to have in the schedule once we account for this being a room full of humans is about ~70 minutes. If we follow through on Tim's proposal, are we really talking two sessions (or three)?

@dr0ptp4kt and @GWicke - is Tim's assessment correct? If so, is this difference something you can resolve before Wikimedia-Developer-Summit-2016? If not, could you explain the situation as you see it (preferably in under 100 words)?

Tim's assessment that we should join the three topics for a 2-3 hour session makes sense to me (@GWicke and I/us have been talking about this sort of stuff for many months now, it's exciting!). @GWicke, we have a recurring 0830 SF weekly recurring meeting for the next several weeks to discuss presentation / breakout - could you make that? If not, you, Jon**2, and I should maybe make a point of talking face to face during better hours over the coming weeks.

@GWicke - just shared the draft slide deck with me. 58 slides?

In a discussion we're having right now (E131), I'm suggesting 20 minutes (maximum) for presentation. Any longer probably needs to be prepared as concisely written pre-briefing rather than having people spend longer than 20 minutes listening to y'all.

There's a more complete agenda now at Wikimedia_Developer_Summit_2016/T114542, which includes several questions for group conversation to capture:

  • Which stakeholders will likely benefit from an API-driven UI approach?
  • Which stakeholders will be negatively impacted by an API-driven UI approach?
  • What alternatives should we consider for API-driven UI?
    • Single-page application
    • ServiceWorker composition
    • (any others?)
  • For each approach, what functionality would need to be reimplemented? Who is responsible for making the change?
  • For each approach, what functionality would need to be dropped? Which stakeholders care?

There are likely other questions that I didn't pull from the slides, but are likely to be asked:

  • Should we load page fragments instead of loading whole pages?
    • Much of the presentation seems to be making the argument for this, but this is likely a contentious point.
  • Do we have to push user-specific page modifications into the browser?
    • Presumably since caching+page-generation gets much easier, but that's likely to be a discussion point.
  • Is the work that WMF planning to do going to make it into MediaWiki, or is this going to be Wikimedia/Wikipedia-specific work?
    • This may be easier to answer in T113210 at 3:40 PM, but some of the conversation will likely need to be explored here

Thanks, @RobLa-WMF - this should give us a good place to start. We should be able to adjust based on the general interests in the room.

dr0ptp4kt updated the task description. (Show Details)Jan 4 2016, 7:59 PM

Wikimedia Developer Summit 2016 ended two weeks ago. This task is still open. If the session in this task took place, please make sure 1) that the session Etherpad notes are linked from this task, 2) that followup tasks for any actions identified have been created and linked from this task, 3) to change the status of this task to "resolved". If this session did not take place, change the task status to "declined". If this task itself has become a well-defined action which is not finished yet, drag and drop this task into the "Work continues after Summit" column on the project workboard. Thank you for your help!

Qgil removed a subscriber: Qgil.Feb 16 2016, 8:08 AM