Page MenuHomePhabricator

Reduce size of module version hash
Closed, ResolvedPublic

Description


The size of the module version hash is currently 7 characters, by base36-encoding the result of fnv132 (as done in ResourceLoader::makeHash).

Contrary to what I thought, this algo usually only needs 6 characters, not 7. As such, for more than half of the modules in production (enwiki/anon/defaults), the base conversion is padding a zero.

let startsWithZero = 0;
let startsWithNonZero = 0;
mw.loader.getModuleNames().forEach(name => {
  mw.loader.getVersion(name)[0] === '0' ? (startsWithZero++) : (startsWithNonZero++);
});
({ startsWithZero, startsWithNonZero })
//> { startsWithZero: 637, startsWithNonZero: 595 } 

About 52% of module hashes have a zero padded. And 3% even has two zeroes padded.

let startsWithTwoZero = 0;
mw.loader.getModuleNames().forEach(name => {
  mw.loader.getVersion(name)[1] === '0' && (startsWithTwoZero++);
});
startsWithTwoZero 
///> 34
Figure 1. Zeroes galore

To do:

  • Document what our needs are of the hashing algo.
  • Reduce the size from 7 to 6.
  • Consider reducing it further.

Event Timeline

Krinkle created this task.Jul 29 2019, 4:13 PM
Restricted Application added a project: Performance-Team. · View Herald TranscriptJul 29 2019, 4:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle triaged this task as Normal priority.Jul 29 2019, 7:53 PM
Krinkle moved this task from Inbox to Accepted: Enhancement on the MediaWiki-ResourceLoader board.
Krinkle claimed this task.Aug 6 2019, 12:23 AM
Krinkle moved this task from Backlog: Small & Maintenance to Doing on the Performance-Team board.

Change 528300 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Purge localStorage blob if last written 30+ days ago

https://gerrit.wikimedia.org/r/528300

Krinkle added a comment.EditedAug 6 2019, 12:55 PM

[mediawiki/core@master] resourceloader: Purge localStorage blob if last written 30+ days ago
https://gerrit.wikimedia.org/r/528300

Reminder: After this is rolled out to prod, remove the $wgResourceLoaderStorageVersion .= assignment from wmf-config, which won't be needed anymore (we'll have effectively changes all version hashes, thus making the old increment obsolete).

Change 526182 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Reduce width of module hash from 7 chars to 5

https://gerrit.wikimedia.org/r/526182

Change 528300 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Purge localStorage blob if last written 30+ days ago

https://gerrit.wikimedia.org/r/528300

Change 526182 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Reduce width of module hash from 7 chars to 5

https://gerrit.wikimedia.org/r/526182

Krinkle closed this task as Resolved.Sep 4 2019, 1:41 AM

Confirmed on Beta Cluster:

> mw.loader.getVersion('jquery')
"1noll"

Compared to prod:

> mw.loader.getVersion('jquery')
"1nollmt"
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)

This improvement has now rolled out to production Wikipedia as part of the MediaWiki 1.34.0-wmf.22 branch.

Read more at https://phabricator.wikimedia.org/phame/live/7/post/175/wikipedia_javascript_initialisation_on_a_budget/

In total, the year-long effort is saving 4.3 Terabytes a day of data bandwidth for our user's page views.

That's awesome! Thanks for coordinating this cross team/project effort!

Krinkle updated the task description. (Show Details)Thu, Sep 19, 10:15 PM