Page MenuHomePhabricator

mw.loader.store should not occupy all of localStorage
Closed, ResolvedPublic

Description

In some cases, MediaWikiModuleStore can be quite large (over 1 MB). This can lead to local storage becoming full and causing random problems like T66716. According to https://support.mozilla.org/en-US/questions/963825 Firefox shares local storage space across all hosts on a single domain name, so this may be especially problematic for Firefox.

In theory, this shouldn't be an issue since people should always have some sort of fall-back behavior if there are problems with local storage, but in the interests of defensive programming, perhaps we should put a cap on the maximum size of MediaWikiModuleStore.


See Also:

Upstream;

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ori added a comment.Jun 4 2015, 11:51 PM

@Mattflaschen The last two solutions sound sensible. Do you have any idea how much that might reduce the average size of the cache by?

On a primed cache on enwiki, the top modules by size for me are:

ModuleSize
ext.visualEditor.core624 kB
oojs-ui179 kB
oojs-ui.styles.icons132 kB
oojs-ui.styles90 kB
mmv.ui.canvasButtons71 kB
ext.visualEditor.mwmeta67 kB
jquery.ui.core66 kB
ext.visualEditor.mwtransclusion65 kB
ext.visualEditor.mwimage65 kB
oojs-ui.styles.icons-editing-advanced58 kB
ext.visualEditor.mwcore52 kB
jquery.ime50 kB
moment48 kB
mmv.ui.metadataPanel47 kB
unicodejs46 kB
ext.visualEditor.icons45 kB
jquery.chosen42 kB
jquery.uls37 kB
ext.visualEditor.mediawiki35 kB
jquery.uls.data33 kB

Change 216014 had a related patch set uploaded (by Ori.livneh):
mw.loader.store: decline to store items > 100 kB

https://gerrit.wikimedia.org/r/216014

Change 216014 merged by jenkins-bot:
mw.loader.store: decline to store items > 100 kB

https://gerrit.wikimedia.org/r/216014

While the last patch may be helpful, it still does not prevent the cache from taking up all available localStorage space.

Krinkle added a comment.EditedJun 9 2015, 8:15 PM

On the topic of using IndexedDB for module store, beware that there is an important difference with localStorage.

localStorage, while having a lower quota by default (~ 5MB), doesn't have a way to expand that quota. Which means when it's full, the API returns a failure code but nothing is shown the user.

IndexedDB, while having a higher quota by default (~ 10-50MB), has a way of extending the quota. It's not clear to me what happens when you reach the quota. It appears that browsers default to asking the user to expand the quota.

For the purpose of caching modules, and for Wikipedia in general, I don't think we should ever allow that to happen. So we may have to somehow manually poll the aggregate size of database to prevent the browser from asking the user any such question.

See http://www.html5rocks.com/en/tutorials/offline/quota-research/

SamB added a subscriber: SamB.Jun 29 2015, 5:45 PM

While the last patch may be helpful, it still does not prevent the cache from taking up all available localStorage space.

Especially on Firefox, where e.g. all of the Wikipedias are sharing quota. (There are a lot of Wikipedias, and there are reasons someone might look at quite a lot of them.)

Also, to clean up after this, you'll need to do something like what's described at https://bugzilla.mozilla.org/show_bug.cgi?id=1064466#c11: have a path on each wiki that will clean up the storage, and also a way to open an iframe to that path on *every* wiki for a given domain ...

SamB added a comment.Jun 29 2015, 9:40 PM

So, um, any tips on how to manually clean up the MediaWikiModuleStore keys from all of the wikipedias on Firefox/Iceweasel, including 31.7.0esr as included in Debian Jessie?

ori added a comment.Jun 29 2015, 9:43 PM

So, um, any tips on how to manually clean up the MediaWikiModuleStore keys from all of the wikipedias on Firefox/Iceweasel, including 31.7.0esr as included in Debian Jessie?

It should not be possible to fill it up now that the patch above has been pushed. You must have gotten into some weird state. Clear localStorage (either by nuking your browser history or by using https://addons.mozilla.org/en-US/firefox/addon/volatile-storage/?src=cb-dl-updated) and see if the problem recurs.

That extension does not seems to be enough, since I still get this on every page on Portuguese Wikipedia using Firefox 38.0:

"Exception in store-localstorage-update:" load.php:177:752
"NS_ERROR_DOM_QUOTA_REACHED: Persistent storage maximum size reached" DOMException [NS_ERROR_DOM_QUOTA_REACHED: "Persistent storage maximum size reached"
code: 1014
nsresult: 0x805303f6
location: https://pt.wikipedia.org/w/load.php?debug=false&lang=pt&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=CZ6SC%2Fzc:175] load.php:177:790
log() load.php:177
handler() load.php:155
jQuery.Callbacks/fire() load.php:45
jQuery.Callbacks/self.fireWith() load.php:46
jQuery.Callbacks/self.fire() load.php:46
mw.track() load.php:155
flush() load.php:175

and running

localStorage.clear();

in the console doesn't help either.

SamB added a comment.Jun 30 2015, 1:39 PM
In T66721#1411737, @ori wrote:

So, um, any tips on how to manually clean up the MediaWikiModuleStore keys from all of the wikipedias on Firefox/Iceweasel, including 31.7.0esr as included in Debian Jessie?

It should not be possible to fill it up now that the patch above has been pushed. You must have gotten into some weird state. Clear localStorage (either by nuking your browser history or by using https://addons.mozilla.org/en-US/firefox/addon/volatile-storage/?src=cb-dl-updated) and see if the problem recurs.

Hmm, I'm still a bit nervous about nuking localStorage entirely, though that certainly would be less tedious than visiting every Wikipedia I've ever loaded in this bowser (including mobile versions) one-by-one and clearing the relevant key from the JS console... (though, that might not be so bad if I used a greasemonkey script to do the actual cleanup for each wiki).

Anyway, this:

Change 216014 merged by jenkins-bot:
mw.loader.store: decline to store items > 100 kB
https://gerrit.wikimedia.org/r/216014

... doesn't sound very promising to me. Just because it only stores items < 100 kB does *not* prevent it from storing many such items, which could add up to a lot of usage on a single Wiki; factor in the fact that subdomains share quota on Firefox, and you could still easily hit 5 MiB, no?

Jdlrobson added a subscriber: Jdlrobson.

Could we use ServiceWorker for this now (see T101731) ?

Could we use ServiceWorker for this now (see T101731) ?

Yes. We created T101732 last month.

Florian added a subscriber: Florian.Aug 7 2015, 8:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2015, 8:49 AM

Yup. I recognise this is an issue but I am saying that this issue mostly impacts power users (especially on Firefox). I'm yet to see this on the mobile site, where gadgets and user scripts are currently not available nor used.

Eh, I don't think that it mostly impact power users, it can impact even occasional editors too.
As an example:

  • Visit https://pl.wikipedia.org/wiki/Ostatnia_droga_Temeraire%27a, which is a pretty generic article on the Polish Wikipedia, as anonymous user. Run JSON.stringify( mw.loader.store ).length to see how much data ResourceLoader puts in localStorage (this is 707573 for me on first page load).
  • Open the MediaViewer, which is not a power-user feature. The length increases to 923750.
  • Open the VisualEditor, which is not a power-user feature either. The length increases to 2008078.

This is already 2 MB, and this is just one wiki and an anonymous user (I'm not sure how the browsers calculate the limit, but if anything, this is less than the number of bytes actually stored). Visit another wiki (say, the same article in English, if you're not satisfied with the content in Polish and speak two languages) and this doubles. Log in (which gets you features like Echo and lets you enable some gadgets or beta features), or visit a third wiki, and you're quickly over the 5 MB limit.

Interesting and thanks for the counter-example.
I've taked to @ori about T96155 and the plan is for me to make this code use the new frontend api. When that's in place we could potentially be a bit more clever about what we are allowed to store e.g. limit number of bytes we allow saving for.

Given that MobileFrontend/Gather are only using this storage mechanism to decide whether to show onboarding tutorials for new features, mostly targeting readers who are not editing or newbies, if localStorage is exhausted not showing these things is not the end of the world.

He7d3r added a comment.EditedSep 8 2015, 1:51 PM

I assume anyone who wants to use localStorage is supposed to use try/catch on every attempt to save a value so that unrelated scripts are still executed? E.g.:
https://pt.wikipedia.org/wiki/MediaWiki:Gadget-fastbuttons.js?diff=43318591
https://meta.wikimedia.org/w/index.php?diff=13512749&oldid=13462279&diffonly=1
https://github.com/brion/MediaWiki-MobileSidebar/pull/5/files?w=1

Generally yes, regardless of this bug, even. localStorage likes to throw
exceptions. (For example, apparently it will be defined but throw exception
on access on Firefox if you disable it in browser settings.)

TheDJ added a comment.Sep 14 2015, 8:30 PM

FYI, i had an issue fixed in WikEd, which was throwing exceptions all over the place, because it wasn't expecting a full localStorage.

That's the problem with this issue. it's infectious....

Ltrlg added a subscriber: Ltrlg.Oct 6 2015, 6:03 PM

Change 263417 had a related patch set uploaded (by Ori.livneh):
Module storage: set stricter storage quotas on Firefox

https://gerrit.wikimedia.org/r/263417

Change 263417 merged by jenkins-bot:
Module storage: set stricter storage quotas on Firefox

https://gerrit.wikimedia.org/r/263417

Nothing has changed here since 2014 -- it was a mistake to enable this feature on Firefox at the time, and it is a mistake to enable it now with a 30KB per-module limit. On my browser at the moment, MediaWikiModuleStore:enwiki is using 655KB, which is hardly an improvement from when the bug was first reported. You only need to visit 7 or 8 wikis to exhaust the quota.

I don't think a <250ms performance improvement justifies randomly breaking the many extensions which use localStorage.

mediawiki.js will keep writing module caches until it hits the quota and fails. The functionality of the site after such a failure depends on how much of the quota is left -- effectively the remainder of the division, 5MB modulo the per-wiki module store size. An unlucky combination could leave a user unable to dismiss dialog boxes or to choose a preferred language.

Some extensions try localStorage first, and fall back to a cookie if it fails. But the MW core does not provide a convenient interface for this, leaving each extension to implement it separately.

ori added a comment.Feb 8 2016, 7:54 PM

Has anyone encountered any bugs since rMW7787a85d5c6f: Module storage: set stricter storage quotas on Firefox went out, just under a month ago?

I continue to get this error on every page view using Opera 12 (obviously, since you only blacklisted Firefox).

He7d3r added a comment.Feb 8 2016, 8:59 PM

I still get this on every page view using Firefox 44.

ori added a comment.Feb 10 2016, 8:15 AM

Ok. I have been in denial about this long enough, I suppose. Let's start by disabling it completely in Firefox and Opera. I am still interested in more reports, because it will help determine where we go from there.

Change 269625 had a related patch set uploaded (by Ori.livneh):
ResourceLoader: disable localStorage cache on FF, Opera

https://gerrit.wikimedia.org/r/269625

Change 269625 merged by jenkins-bot:
resourceloader: Disable localStorage cache on FF, Opera

https://gerrit.wikimedia.org/r/269625

ori closed this task as Resolved.Mar 17 2016, 11:56 PM
ori claimed this task.

No reports for over a month; closing.

Tgr added a subscriber: Tgr.Jul 18 2016, 3:56 PM

From T85923: Anonymous user's decision to disable Media Viewer is forgotten after two weeks: wgMediaViewerOnClick apparently cannot be reliably saved to local storage; used space (as per encodeURIComponent(JSON.stringify(localStorage)).length) is 2600025, quota is (per this test) somewhere between 2600000 and 2700000. The browser is Safari. May be related, although 2600025 is 0b1001111010110001011001 which does not seem like something that would be very close to a quota limit.

Krinkle added a comment.EditedJul 18 2016, 4:58 PM

Replied at T85923#2471543 as it seems about a key disappearing, not a key being unable to be set.

As for the issue reported in this task, it's been resolved. In theory, it will always be possible for the storage quota to be reached, at which points random keys will fail to be set. (This is true for cookies as well.)

Task T101732 tracks moving the module cache to a different interface for that reason.

Krinkle updated the task description. (Show Details)Sep 6 2018, 11:56 PM
phuedx removed a subscriber: phuedx.Sep 7 2018, 8:21 AM

As for the issue reported in this task, it's been resolved. In theory, it will always be

So, what is the appropriate course of action? Given that I'm seeing an elevated level (for FF and Chrome) of:

Exception in store-localstorage-update:
:8080/mw-31-00-elastic/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=0xnodom:177 QuotaExceededError: Failed to execute 'setItem' on 'Storage': Setting the value of 'MediaWikiModuleStore:mw-31-00-elastic' exceeded the quota. DOMException: Failed to execute 'setItem' on 'Storage': Setting the value of 'MediaWikiModuleStore:mw-31-00-elastic' exceeded the quota.
    at flushWrites (http://127.0.0.1:8080/mw-31-00-elastic/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=0xnodom:175:212)
logError @ :8080/mw-31-00-elastic/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=0xnodom:177

Are you seeing that error in Firefox? That shouldn't be possible because MediaWiki self-disables the mw.loader.store functionality on all Firefox versions. And upon viewing a page, any previous use of it is also proactively cleared. The flushWrites code should not be reachable. If you find that it is being reached in Firefox, please file a separate task and we'll investigate to make sure it remains off.

As for Chrome, there shouldn't be a course of action needed. The system is meant to be self-correcting. The errors you're seeing are "caught", that means the MediaWiki code did not crash or stop unexpectedly. The code prints the caught error to the developer console as a "warning" so that when developers investigate other issues, they also know this particular condition was reached. Are you finding any loss of functionality in Chrome?

The main problem found in this task was due to multiple wikis storing their modules under the same quota in Firefox. That is not something we can control. More-over, the browser also does not provide a way to let module storage expire from wikis not recently visited. For this reason, we have decided to disable this performance feature in Firefox. In Chrome this should not be a problem given that Chrome sets the quota at the full domain level (not the parent domain).

I noticed in your example that you might be experiencing problems from serving multiple MediaWiki installations from the same domain under different directories. If that is the case and you intend to also offer multiple installations this way in production, I recommend $wgResourceLoaderStorageEnabled = false;.

Alternatively, if this is only for local development, run localStorage.clear(); from the console to clear the storage used by previous/other wiki installations.

This issue seems to be resolved in Firefox 69 and apparently the quota is now per origin. See: https://bugzilla.mozilla.org/show_bug.cgi?id=1064466#c18

So maybe we can start thinking about enabling local storage for Firefox again? At least dependent on the Firefox version number in the user agent?

Thank you, would you mind creating a new task for that?