Page MenuHomePhabricator

Reduce size of module version hash
Closed, ResolvedPublic

Assigned To
Authored By
Krinkle
Jul 29 2019, 4:13 PM
Referenced Files
F30377096: grafana-wmf22.png
Sep 17 2019, 3:33 AM
F30377766: wmchart.png
Sep 17 2019, 3:33 AM
F30378820: grafana-wmf22-pie.png
Sep 17 2019, 3:33 AM
F29898894: capture.png
Jul 29 2019, 4:13 PM
Tokens
"Stroopwafel" token, awarded by Jdlrobson."Party Time" token, awarded by Rosalie_WMDE.

Description


The size of the module version hash is currently 7 characters, by base36-encoding the result of fnv132 (as done in ResourceLoader::makeHash).

Contrary to what I thought, this algo usually only needs 6 characters, not 7. As such, for more than half of the modules in production (enwiki/anon/defaults), the base conversion is padding a zero.

let startsWithZero = 0;
let startsWithNonZero = 0;
mw.loader.getModuleNames().forEach(name => {
  mw.loader.getVersion(name)[0] === '0' ? (startsWithZero++) : (startsWithNonZero++);
});
({ startsWithZero, startsWithNonZero })
//> { startsWithZero: 637, startsWithNonZero: 595 } 

About 52% of module hashes have a zero padded. And 3% even has two zeroes padded.

let startsWithTwoZero = 0;
mw.loader.getModuleNames().forEach(name => {
  mw.loader.getVersion(name)[1] === '0' && (startsWithTwoZero++);
});
startsWithTwoZero 
///> 34
capture.png (1×2 px, 596 KB)
Figure 1. Zeroes galore

To do:

  • Document what our needs are of the hashing algo.
  • Reduce the size from 7 to 5.
  • Consider reducing it further.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle triaged this task as Medium priority.Jul 29 2019, 7:53 PM
Krinkle moved this task from Inbox to Accepted Enhancement on the MediaWiki-ResourceLoader board.

Change 528300 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Purge localStorage blob if last written 30+ days ago

https://gerrit.wikimedia.org/r/528300

[mediawiki/core@master] resourceloader: Purge localStorage blob if last written 30+ days ago
https://gerrit.wikimedia.org/r/528300

Reminder: After this is rolled out to prod, remove the $wgResourceLoaderStorageVersion .= assignment from wmf-config, which won't be needed anymore (we'll have effectively changes all version hashes, thus making the old increment obsolete).

Change 526182 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Reduce width of module hash from 7 chars to 5

https://gerrit.wikimedia.org/r/526182

Change 528300 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Purge localStorage blob if last written 30+ days ago

https://gerrit.wikimedia.org/r/528300

Change 526182 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Reduce width of module hash from 7 chars to 5

https://gerrit.wikimedia.org/r/526182

Confirmed on Beta Cluster:

> mw.loader.getVersion('jquery')
"1noll"

Compared to prod:

> mw.loader.getVersion('jquery')
"1nollmt"

This improvement has now rolled out to production Wikipedia as part of the MediaWiki 1.34.0-wmf.22 branch.

Read more at https://phabricator.wikimedia.org/phame/live/7/post/175/wikipedia_javascript_initialisation_on_a_budget/

In total, the year-long effort is saving 4.3 Terabytes a day of data bandwidth for our user's page views.

That's awesome! Thanks for coordinating this cross team/project effort!