Page MenuHomePhabricator

Refactor ResourceLoader versioning system to use hashes instead of timestamps
Closed, ResolvedPublic

Description

Right now we standardise on timestamps.

For backends that are not timestamp based, I provided a workaround that uses Memcached to observe changes in the hash and provide that moment in time as the timestamp.

Let's invert this and instead use hashes as standard. Where needed we can trivially hash the timestamp value (trivial server-side, not client-side, but we won't need it client-side).

The only thing we need to do client-side is the equivalent of max(). Basically a cross-browser deterministic way of combining hashes into something new that is ideally not much longer than a single 7-10 character SHA-1 substring would be. Some basic function that does something like Java's String.hashCode and combines it into a new value should do the trick.

Advantages:

  • No more cache bloat to "observe" changes. This has more than once caused outages or high amounts of cache misses when we store the hash wrongly or under the wrong fragmentation context.
  • No more ResourceLoader deployment bugs where changes don't propagate through cache if a module A's 10AM change is deployed after module B's 11AM change (due to max() remaining the same).

The latter includes situations like EventLogging using a fake timestamp of epoch+revisionid. Which is ever incrementing properly, but does not play well together with other timestamps.

Related Objects

Event Timeline

Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added subscribers: Krinkle, Catrope, ori.
Krinkle set Security to None.
Aklapper added a project: Technical-Debt.

Change 207661 had a related patch set uploaded (by Krinkle):
[WIP] resourceloader: Implement version hashes instead of timestamps

https://gerrit.wikimedia.org/r/207661

Change 207661 merged by jenkins-bot:
resourceloader: Replace timestamp system with version hashing

https://gerrit.wikimedia.org/r/207661

Krinkle raised the priority of this task from Low to Medium.
Krinkle moved this task from Accepted Enhancement to Assigned on the MediaWiki-ResourceLoader board.
Krinkle removed a project: Technical-Debt.

Change 215631 had a related patch set uploaded (by Krinkle):
VisualEditorDataModule: Use definition summary instead of time tracking

https://gerrit.wikimedia.org/r/215631

Change 215382 had a related patch set uploaded (by Krinkle):
resourceloader: Remove redundant getModifiedTime implementation

https://gerrit.wikimedia.org/r/215382

Change 215381 had a related patch set uploaded (by Krinkle):
resourceloader: Remove redundant getModifiedTime implementations

https://gerrit.wikimedia.org/r/215381

Change 215384 had a related patch set uploaded (by Krinkle):
resourceloader: Remove redundant getModifiedTime implementation

https://gerrit.wikimedia.org/r/215384

Change 215391 had a related patch set uploaded (by Krinkle):
resourceloader: Remove redundant getModifiedTime implementation

https://gerrit.wikimedia.org/r/215391

Change 215384 merged by jenkins-bot:
Remove redundant ResourceLoaderModule::getModifiedTime implementation

https://gerrit.wikimedia.org/r/215384

Change 215381 merged by jenkins-bot:
resourceloader: Remove redundant getModifiedTime implementations

https://gerrit.wikimedia.org/r/215381

Change 215382 merged by jenkins-bot:
Remove redundant ResourceLoaderModule::getModifiedTime implementation

https://gerrit.wikimedia.org/r/215382

The main version system is hash-based. Most internal values still use timestamps but they're all hashed before being exposed.

Further progress tracked via T98087.

Change 215391 merged by jenkins-bot:
Remove redundant ResourceLoaderModule::getModifiedTime implementation

https://gerrit.wikimedia.org/r/215391

Change 215631 merged by jenkins-bot:
VisualEditorDataModule: Use definition summary instead of time tracking

https://gerrit.wikimedia.org/r/215631

Change 437378 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Remove support for Module::getModifiedTime() and getModifiedHash()

https://gerrit.wikimedia.org/r/437378

Change 437378 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Remove support for Module::getModifiedTime() and getModifiedHash()

https://gerrit.wikimedia.org/r/437378