Make top queue fully asynchronous
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Jul 30 2015, 1:06 AM

Description

📘 See also Perf Matters at Wikipedia on our blog.

We should get rid of this last remaining synchronous part of ResourceLoader (aside from the stylesheet request).

Performance metrics, timelines and waterfalls clearly identify our blocking/synchronous top queue (startup module and the first dynamic load request) as a major pitfall blocking the first paint.

This is especially bad considering our pages are actually fully built server-side. The rare few exceptions to this are isolated to run-time features or enhancements based on user-interaction. Not the first paint.

Tracking:

Move handling of client-nojs/client-js to a new place (embedded in HTML instead of in a top-loading module).
- Also requires it to be refactored to work without access to isCompatible from the startup module. Let's swap the class unconditionally, and add it back at runtime if needed (applies to browser that are unsupported but support some level of JavaScript).
Make modules=startup request asynchronous.
- Refactor base modules request inside startup (for jquery/mediawiki) to use DOM (async) instead of document.write.
- Refactor embedded script tags to buffer their statements into a queue instead of running immediately.
Make top mw.loader.load() queue asynchronous.
- Remove synchronous handling from mw.loader class.
T102077: Load legacy modules "site" and "user" asynchronously.
- Ensure "user" will still execute after "site".
Disabling "legacy" gadgets (T75810: Require all gadgets to load via ResourceLoader):
- https://gerrit.wikimedia.org/r/#/c/230014/

Details

Subject	Repo	Branch	Lines +/-
resourceloader: Async all the way	mediawiki/core	master	+150 -183
resourceloader: Async all the way	mediawiki/core	wmf/1.26wmf17	+150 -183
resourceloader: Convert inline statements to queued functions	mediawiki/core	master	+37 -31

Customize query in gerrit

Revisions and Commits

rMW MediaWiki
	rMW68538086b30f Updated mediawiki/core Project: mediawiki/extensions/Gadgets…
rMEXT MediaWiki Extensions
	rMEXT62e612bbf670 Updated mediawiki/extensions Project: mediawiki/extensions/Gadgets…
rEGAD extension-Gadgets
	rEGAD29e7a9193d78 Don't load legacy gadget scripts
	rEGAD268d49a8c604 Don't load legacy gadget scripts

Related Objects
Search...

Status	Assigned	Task
Resolved	Krinkle	T107399 Make top queue fully asynchronous
Resolved	Krinkle	T102077 Load bottom script "site" and "user" asynchronously
Resolved	Krinkle	T32358 Support loading legacy "site" and "user" modules with mw.loader
Resolved	Krinkle	T75810 Require all gadgets to load via ResourceLoader
Declined	Krinkle	T110749 Load all gadgets via ResourceLoader regardless of definition

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 227627 merged by jenkins-bot:
resourceloader: Async all the way

https://gerrit.wikimedia.org/r/227627

ori mentioned this in rMWd7905627fdc3: resourceloader: Async all the way.Aug 4 2015, 5:30 PM

• Forrestbot added projects: MW-1.26-release, WMF-deploy-2015-08-11_(1.26wmf18).Aug 4 2015, 6:00 PM

Krinkle closed this task as Resolved.Aug 4 2015, 11:30 PM

Jdlrobson mentioned this in T104362: Certain tests failing in bypassed target mode.Aug 5 2015, 12:36 AM

Krinkle mentioned this in T102718: mw.user.options has only default values in debug mode (because user.options module code incorrectly executes before user.defaults).Aug 5 2015, 1:00 AM

Krinkle mentioned this in T92466: For raw script loading, use document.write() to get a callback for the previous document.write().Aug 5 2015, 1:06 AM

Krinkle mentioned this in T39894: Turn startUp() function into a local callback for mediawiki.js in startup.js.Aug 5 2015, 1:43 AM

Krinkle mentioned this in T107976: Graph must not use synchronous `window.mw` check (doesn't work asynchronous).

Krinkle mentioned this in T105872: ResourceLoader dependencies are not always reliable in debug mode.Aug 5 2015, 1:46 AM

Krinkle mentioned this in T48928: On complex installations with many custom JS resources, some Javascript files fail to load.

Krinkle mentioned this in T106736: "Site" and "user" module (legacy only=scripts requests) stay pending indefinitely if script causes error.Aug 5 2015, 1:53 AM

Krinkle mentioned this in T76204: Enforce JavaScript syntax check when editing user/site script pages.Aug 5 2015, 7:11 PM

Krinkle mentioned this in T108124: [Regression] ResourceLoader no longer ensures presence of mw.util and wikibits before site/user modules.Aug 5 2015, 9:51 PM

Issues uncovered so far:

T107979 – VE load order had a race condition that was much more common with async; fixed thank to Krinkle/Roan.
T107980 – Above-the-fold FOUC for VE edit tab (showstopper in my POV)
T107968 – Looks/looked like a mw.EmbedPlayer issue but was being masked by T107979; now seems gone, but no merges in the past few hours?
unfiled – At least twice VE gave the 'anon' notice (so mw.user.isAnon() presumably returned false) despite depending on mediawiki.user; not yet reliably reproducible.
unfiled – Dozens of times, VE causing Chrome to crash tab (unrelated?)
unfiled – RESTbase fetch via load.php seems to be failing at times (unrelated?)

Florian subscribed.Aug 6 2015, 5:17 AM

PerfektesChaos subscribed.Aug 6 2015, 9:19 AM

TheDJ subscribed.Aug 6 2015, 11:16 AM

MrStradivarius subscribed.Aug 6 2015, 3:51 PM

Change 229994 had a related patch set uploaded (by Krinkle):
resourceloader: Async all the way

https://gerrit.wikimedia.org/r/229994

Change 229994 merged by Ori.livneh:
resourceloader: Async all the way

https://gerrit.wikimedia.org/r/229994

ori mentioned this in rMWc5c52ec1d8a2: resourceloader: Async all the way.Aug 6 2015, 11:21 PM

• Forrestbot added a project: WMF-deploy-2015-08-04_(1.26wmf17).Aug 7 2015, 12:00 AM

Krinkle added a commit: rMW68538086b30f: Updated mediawiki/core Project: mediawiki/extensions/Gadgets….Aug 7 2015, 1:08 AM

Krinkle added a commit: rMEXT62e612bbf670: Updated mediawiki/extensions Project: mediawiki/extensions/Gadgets….Aug 7 2015, 1:12 AM

ori added a commit: rEGAD268d49a8c604: Don't load legacy gadget scripts.Aug 7 2015, 1:13 AM

Krinkle added a commit: rEGAD29e7a9193d78: Don't load legacy gadget scripts.Aug 7 2015, 1:13 AM

He7d3r updated the task description. (Show Details)Aug 7 2015, 2:14 AM

He7d3r added a project: User-notice.

Florian awarded a token.Aug 7 2015, 5:37 AM

Hi,

I've seen anouncements on Wikitech-ambassadors about the document.write breaking change, but nothing about legacy gadgets which as far as the French Wikipedia is concerned breaks much more things.

Sorry if I missed it, but now there seems to be a lot of stuff to unbreak there, since about 30 gadgets have never been ported to ResourceLoader.

Anyway, congratulations for the improvements!

Seems to have caused T108359 (broken iframe embedding for audio/video files)... I can fix it using window.RLQ directly but that feels dirty... proper fix is to rewrite the iframe embedding entirely to not be weird and scary, I'll probably do that later. :)

gpaumier moved this task from To Triage to Announce in next Tech/News on the User-notice board.Aug 7 2015, 9:42 PM

gpaumier moved this task from Announce in next Tech/News to In current Tech/News draft on the User-notice board.Aug 7 2015, 9:52 PM

gpaumier moved this task from In current Tech/News draft to Recently announced in Tech/News on the User-notice board.Aug 7 2015, 10:22 PM

PerfektesChaos mentioned this in T108157: Sentry running into infinite loop on beta.wmflabs.org.Aug 8 2015, 12:09 PM

Another graph (screenshot from https://grafana.wikimedia.org/#/dashboard/db/reading-usertype ):

TIme to first paint (Desktop v. Mobile) [anon] [median] 2015-07-24..2015-08-10.png (293×756 px, 60 KB)

gpaumier awarded a token.Aug 10 2015, 10:27 PM

TheDJ mentioned this in T108513: [1.26-wmf17 Regression] Dismissed site notice flashes briefly on page load.Aug 11 2015, 8:28 PM

TheDJ mentioned this in T108811: DismissableSiteNotice uses document.write.Aug 12 2015, 11:37 AM

Johan moved this task from Recently announced in Tech/News to In current Tech/News draft on the User-notice board.Aug 13 2015, 6:52 PM

PleaseStand mentioned this in T109045: Missing tab styles in "DB clusters" page (on noc.wikimedia.org).Aug 14 2015, 9:51 AM

He7d3r updated the task description. (Show Details)Aug 15 2015, 1:54 PM

He7d3r added a subtask: T75810: Require all gadgets to load via ResourceLoader.

Aklapper mentioned this in T109185: Javascript gadget to fill Edit Summary in Tamil Wikipedia is not working.Aug 17 2015, 10:47 AM

Dineshkumar subscribed.Aug 17 2015, 10:51 AM

Johan moved this task from In current Tech/News draft to Recently announced in Tech/News on the User-notice board.Aug 20 2015, 8:52 AM

Johan moved this task from Recently announced in Tech/News to Already announced/Archive on the User-notice board.

Krinkle mentioned this in T109837: Remove top/bottom queue distinction.Aug 21 2015, 3:30 PM

TheDJ mentioned this in T110501: redirectToFragment script (redirects to section headers) behaving unreliably.Aug 27 2015, 9:09 AM

Krinkle mentioned this in T109259: TypeError: $.collapsibleTabs.instances.each is not a function.Aug 27 2015, 11:13 PM

mwjames mentioned this in rESMW9d0118e6ccb7: MW 1.26 / T107399 / Async RL causes style delay.Sep 2 2015, 8:09 PM

mwjames mentioned this in rESBL727fbc38a7f9: MW 1.26 / T107399 / Async RL causes style delay.Sep 15 2015, 11:33 AM

saper mentioned this in T115692: Flash of unstyled content (FOUC) on Login page and Preferences (and possibly other highly visible pages?).Oct 20 2015, 6:18 PM

Can we measure how long does it take for a styled page to appear (and the difference between this and the "first to display" could be the FOUC factor)?

Apart from T115692 I have seen reports that it takes a very long time to load and style the page right after the first install and also for a first time when visiting an previously not visited wiki.

saper mentioned this in T112047: Special:ManageTranslatorSandbox fails to finish loading.Nov 29 2015, 1:09 PM

saper mentioned this in T111685: Unstable interface bug in the Translate form .

saper awarded a token.Nov 29 2015, 1:23 PM

For posterity there was a tweet and an article on observer.com about this project:

Also https://twitter.com/Wikipedia/status/630885153519177728 which got featured in tech talks at various conferences over the past few months (incl. Velocity 2015 New York, and Chrome Dev Summit 2015).