See https://m.mediawiki.org/wiki/Reading/Web/Projects/Barack_Obama_in_under_15_seconds_on_2G for further supporting information.
See https://www.mediawiki.org/wiki/Wikimedia_Engineering/2015-16_Q3_Goals#Reading for Reading goals for Q3 FY 2015-2016 (January - March 2016).
## Acceptance criteria
[ ] Reduce data usage in mobile web beta by 20%
[ ] Reduce first paint / time to interact to 5 seconds or less at the median article size and to 10 seconds or less at the 90th percentile article size on simulated / controlled 2G connections (lag excluded) on mobile web beta.
[ ] Continues to support Wikipedia Zero
##Description
During Q2 FY2015-2016 (October - December, 2015), the WMF Reading Web team conducted R&D on speed improvements with a Node.js driven composition layer and ServiceWorker, along the lines of {T111588} and {T106099}. This was to get a handle on addressing problems exemplified by things like the Barack Obama article on a 2G connection rendering [[ http://www.webpagetest.org/result/150908_J5_14NT/ | lengthily at 50 seconds ]].
By shrinking the HTML to just show the first section of the article and delay rendering of images we can [[ http://www.webpagetest.org/result/150922_K4_250/ | get that down to 11s ]].
It thus follows that with some radical changes to the HTML we serve our users we should be able to get an article loading on a 2G connection much faster. This would be amplified through use of ServiceWorker, but we are not planning to implement ServiceWorker in Q3 just yet (although maybe later), based on feedback at the [[ https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016 | Wikimedia Developer Summit ]] in early January 2016 (see {T114542} for history).
**This** task is a Q3 goal to apply the findings from Q2 and the discussion at the summit to the mobile web beta channel in Q3 (January - March 2016). Q4 is the target for subsequently progressively rolling this into the mobile web stable channel..
Google [[ http://googlewebmastercentral.blogspot.com/2015/04/faster-and-lighter-mobile-web-pages-for.html | already optimises our pages ]] and they shouldn't have to do that . Their data shows speed enhancement resulted in a 50% increase in traffic (as well as a 80% reduction in bytes). We would expect similar numbers to be reflected in our changes.
## Objective
Make Wikipedia more accessible to 2G connections. We plan to implement a web experience in mobile web beta that leverages service oriented architecture (SOA) and accommodates slow (e.g., 2G or congested networks) and unreliable (e.g., mobile cellular) connections without introducing unnecessary drag on Wikimedia origin server and edge cache infrastructure.
## Rationale
The website is unacceptably slow on slow connections. The Global South, where slow connections are more prevalent, is the core strategic focus of the Reading Web team. The necessary new architecture happens to be a great opportunity to employ server resources more judiciously and position us for faster innovation in the future, too.
## Key result
The new experience will
* Reduce data usage in mobile web beta by 20%
** Drive down bandwidth consumption of in-article raster images. For <noscript> browsers requiring a tap through to view images may be preferable to displaying images in-article; for JavaScript supporting, but jQuery-ResourceLoader incompatible browsers <script> tags may be advisable.
** Drive down content size and network round trips for the common cases.
* Reduce first paint / time to interact to 5 seconds or less at the median article size and to 10 seconds or less at the 90th percentile article size on simulated / controlled 2G connections (lag excluded) on mobile web beta.
* Attempt to drive down full article load (non-image media) with pages of median size on 2G connections to [[ https://www.mediawiki.org/wiki/Reading/Web/Projects/Barack_Obama_in_under_15_seconds_on_2G | under 15 seconds ]] plus network lag time.
* If time permits, on supported browsers, when a connection is unavailable upon back/forward navigation to a previously retrieved page, not result in an error message.
* If possible, stop fragmenting the cache on a per-user, per-article basis.
##Dependencies
* Potentially the Security team for security review of associated components
* Architecture team for consultation
* Performance team for patch consultation and review
* Services and Parsing teams for service layer concerns on an as needed basis
* Potentially Analytics team for review of updated pageview and other impression-level data as required
Additionally:
* Technical Operations patch review support will likely be needed for VCL or Apache configuration file changeset review.
* Release Engineering may need to be consulted in an hoc fashion for consultation on unit/TDD test approach and deployment process
* Partnerships may need to facilitate discussions with search and OS providers indexing content, and should continue to spot check Wikipedia Zero compliance occasionally.
##Assumptions/risks
* Wikipedia Zero support will remain intact. In practice, this likely means a continued cache split across W0 on/off status in order to support <noscript> UAs (n.b., inlined <script> is also used in W0 to support ResourceLoader-impaired devices).
* High performance APIs
** References Ajax endpoint
** References HTML page endpoint for RL-incompatible UAs.
** Reference subscript lookup
** Link preview (generic service already exists, but will need a few tweaks) if deemed appropriate.
* Potentially the ability to split origin side parser cache objects; although RESTbase objects with edge caching may achieve similar means if origin side parser cache object splits in PHP are not realistic.
* Some api.php endpoints will likely out of short term convenience need to be consumed directly by Ajax supporting browsers even though in the future optimized RESTbase services may make more sense. Others should likely be fronted by RESTbase to get the performance and simplicity benefits.
* SEO must be accounted for. Formally speaking, mobile web beta pages should be (and if deemed necessary, explicitly must be) considered non-indexable, but the architecture must be built in a manner that will support proper search engine indexing. This likely entails a review of existing known popular search engine documented behaviors in available regions plus consultation with a key set of search engine and OS providers in the Global North and a small set of search engine providers in the Global South (particularly in Eurasia). More specifically, semantics or implicit behaviors must be in force whereby incremental loading doesn't result in content becoming effectively invisible in search engine graphs.
* For Q3, the locus of view controller logic will continue to be MobileFrontend via conventional MediaWiki PHP, ResourceLoader (RL), and also additionally most likely some enhancements to RL/ESI/both involving inlined presentational items like CSS and JS.
* <noscript> and jQuery-incompatible browsers do not need every JavaScript feature emulated at the server, but they do need sensible fallbacks, particularly those that support the core reading experience. As a general rule simpler is better both for the user and the software engineering team for such devices, but there's usually a bit of hard work in this space.
##Future considerations
* [[ https://infrequently.org/2015/06/progressive-apps-escaping-tabs-without-losing-our-soul/ | Progressive web apps ]] and other packaging may make sense in a future state depending on user desire or perceived user benefit for home screen placement and servicing the needs of different users (e.g., desktop packaged apps).
* Throughout Q3 and Q4 learnings should be gathered for consideration for the desktop form factor, where there is a much more complex ecosystem of gadgets.
* This is really out of scope for Reading Web, but cannot be ignored and should instead be considered and planned: VisualEditor is already a sophisticated JavaScript system reliant upon Parsoid/RESTbase. At last check, the plan was to replace the exiting MobileFrontend-integrated VisualEditor with something better tied directly into the VisualEditor codebase. To date, the fuller VisualEditor software architecture is said to presuppose a fuller Parsoid markup than what might be desirable in order to be bootstrapped quickly when a user chooses to initiate editing on a longer document, especially on mobile devices. If the current reading DOM already had all the markup, part of this bootstrapping cost could likely be avoided. But downloading the full Parsoid output on mobile devices is questionable; alternative approaches may include: (1) when there is a strong signal that the user may be predisposed to edit (e.g., logged in with edits), for reading actually get the full Parsoid markup when the connection profile and device/UA characteristics strongly signal the reading experience will still be performant (but do apply transforms to simplify the layout for reading anyway), (2) inform the user when transitioning into editing mode that it will take a while to load (and offer a graceful cancel button in case loading is actually taking too long), or (3) adapt VE to lighter weight markup, probably on some sort of section-by-section basis (there are a number of technical challenges and UX tradeoffs that follow). It would be interesting to consider the notion of a a data savings toggle setting, which may play nicely into #1 and #2; for active editors who activate they're good to go. Additionally, there are alternative/complimentary contributory models, many of which now can even be done in newer devices with ease, perhaps even as JavaScript web applications, although packaging to free up screen real estate may be necessary for the greatest impact:
** Smaller targeted tasks may be better for many form factors, particularly really small ones.
** Touch interface-oriented tasks may also be better suited to various mobile form factors (e.g., object relational modeling or gesture based moderation, subselection, and filtering, tasks).
** And there's always <textarea> to cut out the overhead and for certain types of users who don't mind pinching and zooming as long as their typing and selection cursors will work predictably.
##Future future considerations
The Q3 work is not as dramatic as the original R&D and summit proposals. But we should start considering potential further enhancements.
For example, for non-JavaScript users we could simply serve a "Read full article" link at the bottom of the page that when clicked would load the page via a query string parameter, such as fullarticle=1. To do this we would most likely need to partner with the Operations and Parsing teams since this is likely to fragment our cache / require us caching two versions of the article on mobile along with the PHP desktop cache. In this case we would likely need to:
# Switch to Parsoid as the parser for the mobile skin from the PHP Parser.
# Aggressively reduce HTML sent down the wire and use the Parsoid API to defer load it.
# Further grow performance testing infrastructure to be able to demonstrate the difference
# Optimise further page visits by making use of client side ServiceWorkers to minimise the data we need to send down the wire.
In this future state:
* Minimally, there would need to be a content service layer (RESTbase/Node.js) with key endpoints that exhibit the properties of performant edge caching while observing origin side Parser initiated purges. For example:
** The first part of an article. The object should be retrievable in both a full Parsoid output representation and a more bandwidth conserving representation.
** The other part of an article. The object should be retrievable in both a full Parsoid output representation and a more bandwidth conserving representation.
* There may need to be a setting available to a user labeled "Data savings" that is default on, but when turned off would use fuller HTML with less late bound content stitching, for example, it would:
** Return to the legacy image downloading techniques (e.g., not downsampled, srcset bearing, downloaded immediately instead of in a lazy fashion for <noscript> browsers and immediately after first article portion for JavaScript capable browsers)
** Obtain full Parsoid HTML (probably with styles that are sufficient for larger mobile form factors and higher, but that don't remove as much of the "unnecessary stuff")
* In the future the locus of control for view controller logic would more ideally be or less exclusively by a framework like React or via a homebrewed ServiceWorker composition. Even in this case there would likely be at least some level of hybrid usage in the short term. As but one example, the PHP-based MediaWiki i18n subsystem is probably best left as it is: a PHP-based MediaWiki i18n subsystem that can be consumed on demand (either by middleware or a browser). And yet other composition tasks may be better facilitated via server-side ServiceWorker (as referenced earlier, see discussion at T106099).
* The ability to fully deploy this more dramatic sort of change to the stable channel in a subsequent quarter may hinge in part on data center replication (i.e., partial rollout to the stable channel may be required instead until there is proper data center replication).