Page MenuHomePhabricator

Prepare MediaWiki for API-driven frontend
Open, MediumPublic

Description

This is a rough sketch to seek input from other developers.

Rationale

Enable MediaWiki to respond to page views in a fairly performant manner for all users (logged-in or not).

Outcomes
  • A site admin can enable static caching (eg. wgUseFileCache) and the cache will be used for both logged-out and logged-in users by having the minimal run-time still apply the skin after FileCache. (It is currently applied before FileCache, and anon-only.)
  • Routing logic will know which urls are page views and which not. (Currently special pages can disable OutputPage mid-way.)
  • The Skin is given (cacheable) data about the skin, the page and the user. It will produce its HTML output without performing additional queries. Therefore, any additional DB queries skins currently make for page-related data must instead happen in a way that is triggered earlier when the page is being cached (e.g. skin hooks for Parser, ParserCache or FileCache?)
Small but impactful changes
  1. MediaWiki will have a router that reliably know how to construct an output handler (e.g. no unpredictable OutputPage::disable() half-way through the request, such as in SpecialExport, RawAction and elsewhere).
  2. FileCache no longer caches entire HTML response. Instead it just stores the core page content and relevant data properties needed by the skin (e.g. display title, last modified date, categories, permissions, etc.).
  3. Skin Template must be programming-language neutral (e.g. Mustache template).
  4. Data flow in one direction only. No more two-way communication between Skin and OutputPage. OutputPage injects all relevant data (maybe some of it lazy-computed) into the Skin class. Skin class may combine it with User info and transform properties (renaming, deriving, etc.), and then invokes the Skin template with it.
  5. Skin hooks must change from being "callables with global context returning uncacheable raw html" to instead be "callables that return cacheable HTML or template partials". They can still vary by user and page (e.g. "What links here", "My contributions"), but the decision must be made in Mustache syntax using only the available user/page information. This way it can be rendered in a different server process, or even client-side, and still work and have all the information (T106099). It also means that on page views, all available meta data is naturally already fetched and cacheable. If extension hooks require additional information, they can do so, but must use hook behind the cache, not in front of it. E.g. to do their query when the page cache blob is generated. Not on-demand for every page view.

Other thoughts:

  • As things unfold, it might make sense to obsolete FileCache in favour of a better ParserOutput cache (if it has all the needed data and gains support for storing on-disk instead of in SQL). Given that the run-time overhead of taking a ParserOutput object and applying the Skin template would be cheap enough that it would be negligible. Alternatively, we may change FileCache to be more like a companion to ParserCache that caches only the extra data needed by skins. – Or store them in regular ObjectCache/mainCache instead.

Steps:

  • Deprecate OutputPage::disable().
  • Provide a router. This will map paths and query parameters to a handler, similar to MediaWiki::parseTitle() and other methods in MediaWiki.php do now. The decision to not use a skin or OutputPage for a response must be made here. The router registry should be in a static programming-language neutral format that can be exposed over the API for potential use by a different server process.
  • Make data flow in one direction from OutputPage to Skin.
  • Convert Skin hooks.
  • [ x] Convert one Skin (e.g. Vector) to become a template, as example. (Note, there is no pressure to remove support for PHP-based skins, the only requirement is that it can handle responses in front of the page cache instead of behind it, e.g. using data properties instead of run-time database queries and WikiPage methods calls).

What this task is not

This task does not change the architecture of MediaWiki or the skin system. It also does not change how MediaWiki is used or deployed at WMF.

This work covered by this task resolves long-standing technical debt in MediaWiki for the purposes of improved rendering performance and reduced overall complexity and maintenance cost for the skin system – whilst remaining fully backwards compatible.

The intended result is improved performance of rendering page views in MediaWiki core, both out-of-the-box and at WMF.

The works covered by this task should in my opinion be considered a prerequisite for the following proposals. Some of the below might be possible without this task, although I believe doing so would incur significant amounts of maintenance overhead and technical debt that can and should be avoided (through this task).

  • Add a way for MediaWiki to render wiki page, actions, and special pages without a skin. – T114596
  • Re-implement MediaWiki skin rendering for WMF-deployed skins in a micro-service that runs near the CDN. – T111588
  • Re-implement MediaWiki skin rendering for WMF-deployed skins in a way that can be run offline and client-side using service workes. – T106099.

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedNone
ResolvedJdlrobson
ResolvedSpikeJdlrobson
ResolvedAmmarpad
ResolvedJdlrobson
ResolvedAddshore
ResolvedJdlrobson
ResolvedAmmarpad
ResolvedPwirth
ResolvedAmmarpad
ResolvedKrinkle
ResolvedKrinkle
ResolvedJdlrobson
ResolvedJdrewniak
ResolvedNiedzielski
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 406312 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/skins/MinervaNeue@master] Remove use of QuickTemplate::setRef

https://gerrit.wikimedia.org/r/406312

Change 406313 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Polyglot@master] Remove use of QuickTemplate::setRef

https://gerrit.wikimedia.org/r/406313

Change 406315 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/StickToThatLanguage@master] Remove use of QuickTemplate::setRef

https://gerrit.wikimedia.org/r/406315

Change 406316 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] skins: Deprecate QuickTemplate::setRef()

https://gerrit.wikimedia.org/r/406316

Krinkle updated the task description. (Show Details)Jan 26 2018, 6:23 PM

Change 406318 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/skins/Vector@master] Remove 'SkinVectorStyleModules' hook

https://gerrit.wikimedia.org/r/406318

Change 406313 merged by jenkins-bot:
[mediawiki/extensions/Polyglot@master] Remove use of QuickTemplate::setRef

https://gerrit.wikimedia.org/r/406313

Change 406315 merged by jenkins-bot:
[mediawiki/extensions/StickToThatLanguage@master] Remove use of QuickTemplate::setRef

https://gerrit.wikimedia.org/r/406315

Change 406312 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Remove use of QuickTemplate::setRef

https://gerrit.wikimedia.org/r/406312

Change 406318 merged by jenkins-bot:
[mediawiki/skins/Vector@master] Remove 'SkinVectorStyleModules' hook

https://gerrit.wikimedia.org/r/406318

Imarlier moved this task from Inbox to Radar on the Performance-Team board.Jan 30 2018, 9:02 PM
Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.

Change 406316 merged by jenkins-bot:
[mediawiki/core@master] skins: Deprecate QuickTemplate::setRef()

https://gerrit.wikimedia.org/r/406316

Krinkle triaged this task as Medium priority.Feb 2 2018, 10:55 PM
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.
Krinkle edited projects, added Performance-Team; removed Performance-Team (Radar).
Krinkle moved this task from Inbox to Backlog: Small & Maintenance on the Performance-Team board.

Change 428856 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] skins: Move default style modules to getDefaultModules

https://gerrit.wikimedia.org/r/428856

Change 430830 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] API: Add ApiParseTest case for 'styles' in getDefaultModules

https://gerrit.wikimedia.org/r/430830

Krinkle updated the task description. (Show Details)May 4 2018, 3:46 AM
Krinkle updated the task description. (Show Details)

Change 430830 merged by jenkins-bot:
[mediawiki/core@master] API: Add ApiParseTest case for 'styles' in getDefaultModules

https://gerrit.wikimedia.org/r/430830

Change 428856 merged by jenkins-bot:
[mediawiki/core@master] skins: Move default style modules to getDefaultModules

https://gerrit.wikimedia.org/r/428856

Nirmos added a subscriber: Nirmos.May 18 2018, 1:41 AM
Izno updated the task description. (Show Details)Sep 8 2018, 9:31 PM

Change 460406 had a related patch set uploaded (by Krinkle; owner: C. Scott Ananian):
[mediawiki/core@master] WIP: hard-deprecate adding modules from BeforePageDisplay hook

https://gerrit.wikimedia.org/r/460406

Change 485998 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/skins/Vector@master] Start extracting rendering from PHP into Mustache

https://gerrit.wikimedia.org/r/485998

Change 485998 merged by jenkins-bot:
[mediawiki/skins/Vector@master] Start extracting rendering from PHP into Mustache

https://gerrit.wikimedia.org/r/485998

Isarra added a subscriber: Isarra.Jan 24 2019, 9:48 PM
Krinkle updated the task description. (Show Details)Jan 26 2019, 6:50 AM

Change 485998 merged by jenkins-bot:
[mediawiki/skins/Vector@master] Start extracting rendering from PHP into Mustache https://gerrit.wikimedia.org/r/485998

(Patch was reverted in https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/486874)

Izno added a subscriber: Izno.Mar 12 2019, 11:36 PM
Krinkle updated the task description. (Show Details)Sep 18 2019, 3:59 PM

Change 544378 had a related patch set uploaded (by Krinkle; owner: Derick Alangi):
[mediawiki/core@master] skins: Deprecate getDynamicStylesheetQuery, makeI18nUrl and makeNSUrl

https://gerrit.wikimedia.org/r/544378

Change 544378 merged by jenkins-bot:
[mediawiki/core@master] skins: Deprecate getDynamicStylesheetQuery, makeI18nUrl and makeNSUrl

https://gerrit.wikimedia.org/r/544378

Akuckartz added a subscriber: Akuckartz.

Convert one Skin (e.g. Vector) to become a template, as example. (Note, there is no pressure to remove support for PHP-based skins, the only requirement is that it can handle responses in front of the page cache instead of behind it, e.g. using data properties instead of run-time database queries and WikiPage methods calls).

@Krinkle is this done? As we go into 1.36 we'll have Vector using SkinMustache and the method Skin::getTemplateData will be providing all the relevant data for rendering the associated template. I'm curious if there's any tangible tasks you'd like to work on over the next year

Make data flow one-direction between OutputPage and Skin.
Convert Skin hooks.

I'm very curious about these 2 bullet points and whether there are any actionables here right now that I can make a case for as we work on desktop refresh.

Krinkle updated the task description. (Show Details)Jul 7 2020, 8:03 PM

Convert one Skin (e.g. Vector) to become a template, as example. […]

@Krinkle is this done? As we go into 1.36 we'll have Vector using SkinMustache and the method Skin::getTemplateData will be providing all the relevant data for rendering the associated template. I'm curious if there's any tangible tasks you'd like to work on over the next year

I think from a high-level this is basically done. Making SkinMustache a stable API and updating Vector to use it is is essential for the mid-long term to reduce maintenance cost for everyone and for other skins to enjoy the same benefits. But for the purpose of this task I think we can consider that part done.

Make data flow one-direction between OutputPage and Skin.
Convert Skin hooks.

I'm very curious about these 2 bullet points and whether there are any actionables here right now that I can make a case for as we work on desktop refresh.

The task description elaborates on these a bit:

Task description:
  1. Data flow in one direction only. No more two-way communication between Skin and OutputPage. OutputPage injects all relevant data (maybe some of it lazy-computed) into the Skin class. Skin class may combine it with User info and transform properties (renaming, deriving, etc.), and then invokes the Skin template with it.
  2. Skin hooks must change from being "callables with global context returning uncacheable raw html" to instead be "callables that return cacheable HTML or template partials". They can still vary by user and page (e.g. "What links here", "My contributions"), but the decision must be made in Mustache syntax using only the available user/page information. This way it can be rendered in a different server process, or even client-side, and still work and have all the information (T106099). It also means that on page views, all available meta data is naturally already fetched and cacheable. If extension hooks require additional information, they can do so, but must use hook behind the cache, not in front of it. E.g. to do their query when the page cache blob is generated. Not on-demand for every page view.

For the data flow issue, you can uncover some actionables by stepping through (e.g. statically by reading code) Skin::execute and looking for any method calls along the way that aren't injected template data and aren't trivial pure-function helper methods.

For each of those things we can can determine:

  • … whether it's needed e.g. perhaps a simpler approach can work. As an example, the skin is currently querying the database for page metadata for Wikipedia:About, Wikipedia:General_disclaimer, and Wikipedia:Contact_us on every page view. It uses this information to decide whether to style the link as a redirect, or with the user's stub threshold, or as red link. This is due to all of LinkRenderer being pulled in. It's a product decision I suppose, but if you (plural) decide that it's okay for these to just always render as standard blue links (as they do today) this would remove a significant obstacle to making skins more stateless. We can't of course take that easy path on everything. There'll be a few hard problems to solve.
  • … whether they can be computed based solely on information already known to OutputPage or cached ParserOutput for the current page, in which case it should be retreived from there directly.
  • … whether it's something we could trivially expose as part of OutputPage and/or precomputed through ParserOutput. For example, right now the Skin performs its own ad-hoc database query to find out when the page was last modified. This seems like a common enough need that the skin shouldn't have to know how to get this. Even if we end up with a mechanism that is fragmented by skin to precompute some skin-speciifc data, this seems like it is common enough not to need to go through that mechanism.
  • … if it's none of the above, then move it its computation logic to a central place within the Skin (sub) class, e.g. a method not unlike prepareQuickTemplate but that returns a simple array for internal use. For example, if lastModified were a unique need custom to Vector, we'd move it to that new method so that it's easier to deal with long-term and to find what these use cases are then port them to something potentially cached on a per-page bases alongside parser cache or something.
Prod added a subscriber: Prod.Wed, Jan 20, 10:30 PM
Izno removed a subscriber: Izno.Wed, Jan 20, 11:10 PM
Izno added a subscriber: Izno.