Page MenuHomePhabricator

Implement a indexedDB-based key value store in core
Open, Needs TriagePublic

Description

Most client side local storage in MW is done via mw.storage and mw.storage.session which provide simple wrappers around localStorage and sessionStorage respectively, adding some commonly require features:

Both storage systems have quotas of about 5-10MiB, which becomes problematic with use cases like ResourceLoader caching (mw.loader.store which regularly stores 1-2MiB), or document autosave (VE needs to store initial Parsoid HTML which can be several MiB, and the user could have several concurrent autosaves at any one time, same applies to wikitext autosave).

For this reason VE is still using sessionStorage (aka tab storage) for autosave, as this is less likely to ever have multiple documents and is typically short-lived.

More recently indexedDB has become widely available with much greater storage limits (typically on the order of GiB), and was used to provide a longer-lived autosave feature in the core wikitext editors e.g. 2010 wikitext editor (mediawiki.editRecovery).

To allow use of indexedDB in more places, this implementation (mediawiki.editRecovery) should be turned into a generic KV store, along the lines of mw.storage (although it will need to be async).

See also
Tasks mentioning indexeddb: https://phabricator.wikimedia.org/search/query/xc7H.gWgabNB/#R

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change #1131783 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/core@master] mediawiki.storage: Add simple key-value store using indexeddb

https://gerrit.wikimedia.org/r/1131783

This could also be used in mw.loader's ResourceLoader cache, which currently uses localStorage and so limits itself to 100k per module.

Switching to indexedDB would allow us to remove the 100k limit.

This could also be used in mw.loader's ResourceLoader cache, which currently uses localStorage and so limits itself to 100k per module.

Switching to indexedDB would allow us to remove the 100k limit.

A quick test locally after loading a page then VE shows quite a few modules that aren't getting cached:

modulebytes
jquery140685
vue159907
oojs-ui-core189727
moment100765
peerjs115922
ext.visualEditor.core964731
ext.visualEditor.mwtransclusion151667
ext.cite.visualEditor111377
ext.CodeMirror.v6.lib592754
ext.CodeMirror.v6157890
codex-styles159157
@wikimedia/codex173152

The main complication here is that mediawiki.loader is part of the startup module - so loads before everything else. In order to use indexedb storage here, we'd either need to inline direct access (like is currently done with localStorage, bypassing mw.storage) or move our indexeddb API into startup.

This could also be used in mw.loader's ResourceLoader cache, which currently uses localStorage and so limits itself to 100k per module.

Switching to indexedDB would allow us to remove the 100k limit.

CacheStorage seems like it would be a better fit for ResourceLoader.

CacheStorage seems like it would be a better fit for ResourceLoader.

That's T101732: Use Service Worker cache when available for ResourceLoader caching instead of LocalStorage, which is where Ed suggested IndexedDB as alternative in 2023 before filing this task.

The prep work for it (T130855: Make mw.loader.store asynchronous) was completed.

Depending on outcome of T390020, T101732 may be worth re-opening.

That's T101732: Use Service Worker cache when available for ResourceLoader caching instead of LocalStorage, which is where Ed suggested IndexedDB as alternative in 2023 before filing this task.

I suggested CacheStorage because I imagined that it would permit more fine-grained eviction, but that doesn't appear to be the case, based on Chrome Web Storage and Quota Concepts. When there is storage pressure, Chrome will delete all quota-managed data from the least recently used origin, that means both IndexedDB and CacheStorage. If you manages to beat Chrome's heuristics and get the persistent flag, the whole origin will be kept forever regardless of its composition. You could have 1 GB of cache cruft and 100 KB of important user data, Chrome won't distinguish, either it's all kept or none of it is kept.

However, localStorage and cookies are separate and aren't evicted by this mechanism.

I see that CacheStorage has no internal LRU list or expiry concept.

So, I retract my suggestion.

@tstarling The appeal for Cache API was, and imho still is, as follows:

  • localStorage has a hard limit of 10MB across all browsers and uses synchronous I/O. This is especially bad on mobile browsers where (when I last tested this a few years ago), browsers would read and parse the entire localStorage for a given origin into memory regardless of which key you read.
  • The Cache API has no defined hard limit, and is asynchronous.

Browsers each have their way of dealing with disk space pressure and how they are willing to occupy together and how much one origin may occupy. Generally though the overal storage quota for an origin (i.e. Cache API, IDB, http cache, etc) is well above 10MB. I was able to store 50MB in window.caches in Firefox on https://example.org without running into a storage quota or required permission grant (code: P95538).

This is not strongly persistent, but the HTTP cache isn't either. I don't think we're relying on mw.loader.store to outlast the HTTP cache. We have it to defragment the HTTP cache and reuse it more effectively. We actually evict mw.loader.store every 30 days to prevent unbounded reuse for any exposed artefacts (per T134368, T58778, and T229245)