Memory leak from Popups on Wikipedia pages
Open, NormalPublic

Description

@JFG reports that he's experiencing a memory leak in Chrome (WebKit?) whenever a Wikipedia page (but only a Wikipedia page) is left open and idle in another tab for a few hours. The process accumulates memory (up to gigabytes) but uses very little CPU while in the background. Clicking on the tab reduces the allocated memory to a more reasonable amount.

JFG noticed this problem shortly after the New Page Patroller user right, which allows access to page curation tool, was added to his account.

He has already tried:

  • removing all user scripts
  • disabling gadgets in Preferences
  • opening pages in ?safemode=1
  • disabling all browser extensions

"An article page that has been loaded, minimized, left untouched, and shares its process with no other page, grows to over a gigabyte of memory usage over a couple hours."

He has a snapshot of memory use in debug mode available.

Developer notes

See T205127#4613124 and T205127#4613193 for a recently written summary. What I know is that users of several different communities have reported through various channels over the past 2-3 months about memory leaks, after several years of having virtually heard nothing about the subject. They could be unrelated reports, but so far they had in common:

  • Wikipedia (English and German).
  • Multiple browsers (Chrome, Opera, possibly others).
  • Desktop (any skin).
  • Logged-in.

This task in particular shows:

  • There may be multiple sources of memory leaks. As such, let's focus this task only on the issue reported by @JFG.
  • The report from @JFG via @Whatamidoing-WMF includes various bits of information as well as a Chrome trace log.
  • This trace log shows direct correlation on an otherwise idle page that every mouse event that triggered significant code in Popups, also had a subsequent increase in memory use without a decrease in memory at any point. Specific function names, context and reproduction steps are found in T205127#4613193.

While there maybe other leaks in other places, I suggest to create separate tasks about each finding so that we can triage, understand and resolve each one individually.

Restricted Application added a project: Growth-Team. · View Herald TranscriptSep 21 2018, 5:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
JFG added a comment.Sep 21 2018, 7:54 PM

Thanks for reporting my issue here. Let me add that only the English Wikipedia suffers from this leak, which is why I suspected my user rights, scripts or gadgets there. To compare, I have kept pages open for several hours on the German, French, Japanese and Russian Wikipedias, as well as Wikidata, on the exact same browser and configuration, and all the non-English pages have remained stable.

JFG added a comment.Sep 21 2018, 8:03 PM

I took a memory profiling excerpt from the JS console, but I can't upload it here. The system complains that "no configured storage engines can store this file." Now what?

I'm ready to do any tests that may be useful to narrow down the issue. Just ping me.

@JFG thanks for taking the time to investigate this problem.

You can send me the memory profiling log at sbisson@wikimedia.org and I'll find a way to make it accessible to everybody.

Have you tried using another browser (Firefox probably) to see whether the problem is related to Google Chrome?

JFG added a comment.Sep 22 2018, 8:45 AM

Have you tried using another browser (Firefox probably) to see whether the problem is related to Google Chrome?

I have not. I've been using Wikipedia with Chrome for years, sometimes with hundreds of open tabs, and never seen this kind of behavior. Sure, it might be some phenomenon that only appears when you combine enwiki with Chrome, but that's a frequent enough combo among users to be taken seriously.

One thing I could try easily is logging out of Wikipedia, then if the issue disappears we will know that it must be linked to my enwiki user settings. Will do that now.

JFG added a comment.Sep 22 2018, 2:38 PM

One thing I could try easily is logging out of Wikipedia, then if the issue disappears we will know that it must be linked to my enwiki user settings. Will do that now.

When logged out, memory usage is lying low and flat. Therefore the issue is related to my enwiki settings. (Things are fine with dewiki, frwiki, jawiki, ruwiki and wikidata.)

Krinkle added a comment.EditedSep 22 2018, 3:12 PM

@JFG Thanks. And you said it did not happen on enwiki/Chrome with safemode=1, which rules out gadgets and such.

It would still be useful to know if it happens in Firefox for you. But, of course this is a serious problem either way.

By process of elimination, I can think of only two causes:

  • JavaScript code from MediaWiki core or an extension specific to your user settings. There are quite a lot. For example, there is a JavaScript module only loaded when you enable “double-click to edit a section”. It might help to narrow this down. Perhaps you could start by sharing the list of modules loaded for you. I’ll explain below how.
  • JavaScript code from the browser or from a browser plugin that is acting strangely for you on this one website and only when logged in. Do you experience the memory problem if you use an Incognito window in Chrome, and log in there?

To see the list of modules loaded on this page, right-click to "View source", then look for a line of code that starts with "mw.loader.load(", until the next closing parenthesis ")".

JFG added a comment.Sep 23 2018, 8:27 PM

@Krinkle: Thanks for your comments and suggestions. I have run a day-long experiment, so that I can confirm the following:

  • A page loaded in an incognito window while logged in does not leak.
  • A page loaded with ?safemode=1 while logged in leaks like a sieve.
  • All my Chrome extensions are disabled except AdBlock Plus. However AdBlock Plus is turned off on the Wikipedia domains.

Here is the list of modules loaded on an enwiki page when I'm logged in:

mw.loader.load("/w/load.php?debug=false\u0026lang=en\u0026modules=user\u0026skin=vector\u0026user=JFG\u0026version=13hdxv8");

mw.loader.load(["ext.cite.a11y","site","mediawiki.page.startup","mediawiki.user","mediawiki.page.ready","jquery.tablesorter","jquery.makeCollapsible","mediawiki.toc","mediawiki.searchSuggest","mediawiki.page.watch.ajax","ext.gadget.confirmationRollback-mobile","ext.gadget.searchFocus","ext.gadget.exlinks","ext.gadget.PrintOptions","ext.gadget.Twinkle","ext.gadget.teahouse","ext.gadget.ReferenceTooltips","ext.gadget.geonotice","ext.gadget.watchlist-notice","ext.gadget.citations","ext.gadget.DotsSyntaxHighlighter","ext.gadget.HotCat","ext.gadget.wikEdDiff","ext.gadget.DRN-wizard","ext.gadget.charinsert","ext.gadget.refToolbar","ext.gadget.extra-toolbar-buttons","ext.gadget.edittop","ext.gadget.switcher","ext.centralauth.centralautologin.clearcookie","mmv.head","mmv.bootstrap.autostart","ext.popups","ext.visualEditor.desktopArticleTarget.init","ext.visualEditor.targetLoader","ext.echo.init","ext.eventLogging.subscriber","ext.wikimediaEvents","ext.wikimediaEvents.loggedin","ext.navigationTiming","ext.uls.eventlogger","ext.uls.init","ext.uls.compactlinks","ext.uls.interface","ext.cx.eventlogging","ext.cx.interlanguagelink","ext.cx.campaigns.contributionsmenu","ext.3d","ext.centralNotice.geoIP","ext.centralNotice.startUp","skins.vector.js"]);

Same page loaded with safemode yields:

mw.loader.load(["ext.cite.a11y","mediawiki.page.startup","mediawiki.user","mediawiki.page.ready","jquery.tablesorter","jquery.makeCollapsible","mediawiki.toc","mediawiki.searchSuggest","mediawiki.page.watch.ajax","ext.centralauth.centralautologin.clearcookie","mmv.head","mmv.bootstrap.autostart","ext.popups","ext.visualEditor.desktopArticleTarget.init","ext.visualEditor.targetLoader","ext.echo.init","ext.eventLogging.subscriber","ext.wikimediaEvents","ext.wikimediaEvents.loggedin","ext.navigationTiming","ext.uls.eventlogger","ext.uls.init","ext.uls.compactlinks","ext.uls.interface","ext.cx.eventlogging","ext.cx.interlanguagelink","ext.cx.campaigns.contributionsmenu","ext.3d","ext.centralNotice.geoIP","ext.centralNotice.startUp","skins.vector.js"]);

If I understand your reasoning, the culprit must be in this reduced module set, because the page still leaks. Let me know what to try next.

JFG added a comment.Sep 23 2018, 8:31 PM

As a potentially helpful addition, here are the modules loaded in incognito mode (no leak):

mw.loader.load("/w/load.php?debug=false\u0026lang=en\u0026modules=user\u0026skin=vector\u0026user=JFG\u0026version=13hdxv8");

mw.loader.load(["site","mediawiki.page.startup","mediawiki.user","mediawiki.page.ready","mediawiki.toc","mediawiki.searchSuggest","mediawiki.page.watch.ajax","ext.gadget.confirmationRollback-mobile","ext.gadget.searchFocus","ext.gadget.exlinks","ext.gadget.PrintOptions","ext.gadget.Twinkle","ext.gadget.teahouse","ext.gadget.ReferenceTooltips","ext.gadget.geonotice","ext.gadget.watchlist-notice","ext.gadget.citations","ext.gadget.DotsSyntaxHighlighter","ext.gadget.HotCat","ext.gadget.wikEdDiff","ext.gadget.DRN-wizard","ext.gadget.charinsert","ext.gadget.refToolbar","ext.gadget.extra-toolbar-buttons","ext.gadget.edittop","ext.gadget.switcher","ext.centralauth.centralautologin.clearcookie","mmv.head","mmv.bootstrap.autostart","ext.popups","ext.visualEditor.desktopArticleTarget.init","ext.visualEditor.targetLoader","ext.echo.init","ext.eventLogging.subscriber","ext.wikimediaEvents","ext.wikimediaEvents.loggedin","ext.navigationTiming","ext.uls.eventlogger","ext.uls.init","ext.uls.compactlinks","ext.uls.interface","ext.cx.eventlogging","ext.cx.interlanguagelink","ext.cx.campaigns.contributionsmenu","ext.3d","ext.centralNotice.geoIP","ext.centralNotice.startUp","skins.vector.js"]);

And here's incognito + safemode:

mw.loader.load(["mediawiki.page.startup","mediawiki.user","mediawiki.page.ready","mediawiki.toc","mediawiki.searchSuggest","mediawiki.page.watch.ajax","ext.centralauth.centralautologin.clearcookie","mmv.head","mmv.bootstrap.autostart","ext.popups","ext.visualEditor.desktopArticleTarget.init","ext.visualEditor.targetLoader","ext.echo.init","ext.eventLogging.subscriber","ext.wikimediaEvents","ext.wikimediaEvents.loggedin","ext.navigationTiming","ext.uls.eventlogger","ext.uls.init","ext.uls.compactlinks","ext.uls.interface","ext.cx.eventlogging","ext.cx.interlanguagelink","ext.cx.campaigns.contributionsmenu","ext.3d","ext.centralNotice.geoIP","ext.centralNotice.startUp","skins.vector.js"]);

Hope this helps.

@JFG So I I understand correctly, it is specific to these circumstances: Google Chrome, en.wikipedia.org, being logged-in, not using Incognito.And the problem does not happen if you use Incognito, or are logged-in, or browse a different wiki. And use of safemode=1 does not make the problem go away.

The aspect that I find most interesting here is "not using Incognito". I suspect that the problem is caused by one of the 31 modules common to your problematic experiences, but only in combination with personal information stored in your Google Chrome browser. Information that, does not get used in Incognito mode. Probably something in your cookies or localStorage.

Can you run the following two pieces of code and share the result? Note that these may contain personal information. I've written the code so that it removes anything I believe can be personal, but check it yourself before sharing. Feel free to change anything you want with alternate examples, just mention which keys you changed (if any).

Cookie code
JSON.stringify($.cookie(),null, 2).replace(/GeoIP.*/g, 'GeoIP": ***').replace(/([":])[0-9a-fA-Z]+([",])/g, '$1*$2');
Cookie result (example)
{
  "centralnotice_hide_wlm2018": "{\"v\":*,\"created\":*,\"reason\":\"close\"}",
  "GeoIP": ***
}
LocalStorage code
ls = {};
for (let key, i = 0; i < localStorage.length; i++) {
  key = localStorage.key(i);
  if (!key.includes('MediaWikiModuleStore')) ls[key] = localStorage.getItem(key);
}
JSON.stringify(ls,null, 2).replace(/([":!])[0-9a-fA-Z!]+([",!])/g, '$1*$2');
LocalStorage result (example)
{
  "CentralNoticeKV|global|buckets": "{\"expiry\":*,\"val\":\"wlm 2018!*!0\"}",
  "CentralNoticeKV|global|impression_diet_POTY20172": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"skippedThisCycle\":*,\"nextCycleStart\":*,\"seenThisCycle\":2}}",
  "uls-previous-language-autonym": "Nederlands",
  "uls-previous-language-code": "nl",
  "wmE-sS--sessionId": "rejected",
  "wmE-sS--sessionIdEndTime": "*"
}
JFG added a comment.Sep 23 2018, 8:58 PM

Comparing the lists, here are the only modules loaded in a leaky situation (safemode) which are not loaded in the non-leaky case (safemode + incognito):

"ext.cite.a11y",
"jquery.tablesorter",
"jquery.makeCollapsible",

That narrows down the possible issues pretty nicely. Now what?

Comparing the lists, here are the only modules loaded in a leaky situation (safemode) which are not loaded in the non-leaky case (safemode + incognito):

"ext.cite.a11y",
"jquery.tablesorter",
"jquery.makeCollapsible",

That narrows down the possible issues pretty nicely. Now what?

Cool. In addition to the cookie/localStorage information. It would also be useful to have an example page where you are able to reproduce the memory leak. Does it happen on all pages for you, or only specific ones? For example, does it happen on very simple and short pages, such as https://en.wikipedia.org/wiki/Kluivert?

JFG added a comment.Sep 23 2018, 9:12 PM

The simplest page I used when conducting long-term tests was https://en.wikipedia.org/wiki/DRG_Class_01. Just a few pictures and infoboxes in there, no tables or exotic templates. Trying on your "Kluivert" stub now, but reporting might take a few hours.

JTannerWMF moved this task from Inbox to To Triage on the Growth-Team board.Sep 24 2018, 1:20 AM
JFG added a comment.Sep 24 2018, 5:47 AM

Confirmed: Kluivert leaks. (With all due respect to the footballers involved…)

JFG added a comment.Sep 24 2018, 6:17 AM

@Krinkle: Here are the requested object dumps, taken in safemode (because that loads less stuff, and still leaks):

  • Cookies:
"{
  "GeoIP": ***
  "enwikimwuser-sessionId": "*",
  "*": "*",
  "uls-previous-language-autonym": "English",
  "*": "wikitext",
  "wikiEditor-0-toolbar-section": "advanced",
  "wikEdDiff": "*"
}"
  • LocalStorage:
"{
  "CentralNoticeKV|category|MuseumdayCH2018|impression_diet": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"waitCount\":*,\"waitUntil\":*,\"waitSeenCount\":1}}",
  "CentralNoticeKV|category|SwissCalforprojects2019|impression_diet": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"skippedThisCycle\":*,\"nextCycleStart\":*,\"seenThisCycle\":1}}",
  "CentralNoticeKV|category|WMNL_WomenTechStorm2_2018|impression_diet": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"skippedThisCycle\":*,\"nextCycleStart\":*,\"seenThisCycle\":10}}",
  "CentralNoticeKV|global|buckets": "{\"expiry\":*,\"val\":\"Swiss Cal for projects 2019!*!0*wlm 2018!*!1*WMNL_WomenTechStorm2_2018!*!0*Sept2018Maintenance!*!0\"}",
  "CentralNoticeKV|global|impression_diet_POTY2017": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"skippedThisCycle\":*,\"nextCycleStart\":*,\"seenThisCycle\":3}}",
  "CentralNoticeKV|global|impression_diet_POTY20172": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"skippedThisCycle\":*,\"nextCycleStart\":*,\"seenThisCycle\":5}}",
  "CentralNoticeKV|global|impression_diet_inspirehealth": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"skippedThisCycle\":*,\"nextCycleStart\":*,\"seenThisCycle\":4}}",
  "CentralNoticeKV|global|impression_diet_privacypolicymay2018": "{\"expiry\":*,\"val\":{\"seenCount\":*,\"skippedThisCycle\":*,\"nextCycleStart\":*,\"seenThisCycle\":3}}",
  "edittoolscharsubset": "*",
  "expandedSections": "{\"Tropical cyclone\":{\"External_links\":1536033805446},\"2018 in spaceflight\":{\"Suborbital_flights\":*,\"Orbital_launches\":1536033859926}}",
  "ext.popups.core.previewCount": "*",
  "mmMetaUserGroups": "{\"*\":{\"permissions\":[\"createaccount\",\"read\",\"edit\",\"createtalk\",\"writeapi\",\"viewmywatchlist\",\"editmywatchlist\",\"viewmyprivateinfo\",\"editmyprivateinfo\",\"editmyoptions\",\"abusefilter-log-detail\",\"centralauth-merge\",\"abusefilter-view\",\"abusefilter-log\",\"vipsscaler-test\"],\"addRemoveGroups\":false},\"user\":{\"permissions\":[\"collectionsaveasuserpage\",\"reupload-own\",\"move-rootuserpages\",\"move-categorypages\",\"read\",\"edit\",\"createpage\",\"createtalk\",\"writeapi\",\"minoredit\",\"editmyusercss\",\"editmyuserjson\",\"editmyuserjs\",\"purge\",\"sendemail\",\"applychangetags\",\"spamblacklistlog\",\"mwoauthmanagemygrants\"],\"addRemoveGroups\":false},\"autoconfirmed\":{\"permissions\":[\"reupload\",\"upload\",\"move\",\"collectionsaveasuserpage\",\"collectionsaveascommunitypage\",\"autoconfirmed\",\"editsemiprotected\",\"movestable\",\"autoreview\",\"skipcaptcha\",\"abusefilter-log-detail\",\"transcode-reset\"],\"addRemoveGroups\":false},\"bot\":{\"permissions\":[\"ipblock-exempt\",\"changetags\",\"extendedconfirmed\",\"noratelimit\",\"bot\",\"autoconfirmed\",\"editsemiprotected\",\"nominornewtalk\",\"autopatrol\",\"suppressredirect\",\"apihighlimits\",\"writeapi\",\"autoreview\",\"skipcaptcha\"],\"addRemoveGroups\":false},\"sysop\":{\"permissions\":[\"templateeditor\",\"changetags\",\"extendedconfirmed\",\"suppressredirect\",\"noratelimit\",\"deleterevision\",\"deletelogentry\",\"editcontentmodel\",\"block\",\"createaccount\",\"delete\",\"deletedhistory\",\"deletedtext\",\"undelete\",\"editinterface\",\"editsitejson\",\"edituserjson\",\"import\",\"move\",\"move-subpages\",\"move-rootuserpages\",\"move-categorypages\",\"patrol\",\"autopatrol\",\"protect\",\"editprotected\",\"rollback\",\"upload\",\"reupload\",\"reupload-shared\",\"unwatchedpages\",\"autoconfirmed\",\"editsemiprotected\",\"ipblock-exempt\",\"blockemail\",\"markbotedits\",\"apihighlimits\",\"browsearchive\",\"movefile\",\"unblockself\",\"mergehistory\",\"managechangetags\",\"deletechangetags\",\"autoreview\",\"stablesettings\",\"movestable\",\"review\",\"abusefilter-revert\",\"abusefilter-view-private\",\"jsonconfig-flush\",\"oathauth-enable\",\"tboverride\",\"titleblacklistlog\",\"transcode-reset\",\"transcode-status\",\"globalblock-whitelist\",\"nuke\",\"skipcaptcha\",\"override-antispoof\",\"abusefilter-log-detail\",\"massmessage\"],\"addRemoveGroups\":true},\"interface-admin\":{\"permissions\":[\"editinterface\",\"editsitecss\",\"editsitejson\",\"editsitejs\",\"editusercss\",\"edituserjson\",\"edituserjs\",\"oathauth-enable\"],\"addRemoveGroups\":false},\"bureaucrat\":{\"permissions\":[\"move-subpages\",\"suppressredirect\",\"tboverride\",\"noratelimit\",\"oathauth-enable\",\"override-antispoof\"],\"addRemoveGroups\":true},\"reviewer\":{\"permissions\":[\"review\",\"autoreview\",\"autoconfirmed\",\"editsemiprotected\"],\"addRemoveGroups\":false},\"steward\":{\"permissions\":[\"noratelimit\",\"userrights\",\"globalblock\",\"centralauth-unmerge\",\"centralauth-lock\",\"centralauth-oversight\"],\"addRemoveGroups\":false},\"accountcreator\":{\"permissions\":[\"override-antispoof\",\"tboverride-account\",\"noratelimit\"],\"addRemoveGroups\":false},\"import\":{\"permissions\":[\"importupload\",\"import\"],\"addRemoveGroups\":false},\"transwiki\":{\"permissions\":[\"import\"],\"addRemoveGroups\":false},\"ipblock-exempt\":{\"permissions\":[\"ipblock-exempt\",\"torunblocked\"],\"addRemoveGroups\":false},\"oversight\":{\"permissions\":[\"browsearchive\",\"deletedhistory\",\"deletedtext\",\"abusefilter-view-private\",\"deleterevision\",\"deletelogentry\",\"hideuser\",\"suppressrevision\",\"suppressionlog\",\"abusefilter-hide-log\",\"abusefilter-hidden-log\",\"oathauth-enable\"],\"addRemoveGroups\":false},\"founder\":{\"permissions\":[\"userrights\"],\"addRemoveGroups\":false},\"rollbacker\":{\"permissions\":[\"rollback\"],\"addRemoveGroups\":false},\"abusefilter-helper\":{\"permissions\":[\"spamblacklistlog\",\"abusefilter-view-private\"],\"addRemoveGroups\":false},\"autoreviewer\":{\"permissions\":[\"autopatrol\"],\"addRemoveGroups\":false},\"eventcoordinator\":{\"permissions\":[\"noratelimit\"],\"addRemoveGroups\":true},\"researcher\":{\"permissions\":[\"browsearchive\",\"deletedhistory\",\"apihighlimits\"],\"addRemoveGroups\":false},\"filemover\":{\"permissions\":[\"movefile\"],\"addRemoveGroups\":false},\"checkuser\":{\"permissions\":[\"browsearchive\",\"deletedhistory\",\"deletedtext\",\"abusefilter-view-private\",\"abusefilter-private\",\"abusefilter-private-log\",\"oathauth-enable\",\"checkuser\",\"checkuser-log\"],\"addRemoveGroups\":false},\"templateeditor\":{\"permissions\":[\"templateeditor\",\"tboverride\"],\"addRemoveGroups\":false},\"massmessage-sender\":{\"permissions\":[\"massmessage\"],\"addRemoveGroups\":false},\"extendedconfirmed\":{\"permissions\":[\"extendedconfirmed\"],\"addRemoveGroups\":false},\"extendedmover\":{\"permissions\":[\"suppressredirect\",\"move-subpages\",\"move\"],\"addRemoveGroups\":false},\"patroller\":{\"permissions\":[\"patrol\"],\"addRemoveGroups\":false},\"abusefilter\":{\"permissions\":[\"abusefilter-modify\",\"changetags\",\"managechangetags\",\"oathauth-enable\"],\"addRemoveGroups\":false},\"epcoordinator\":{\"permissions\":[],\"addRemoveGroups\":true},\"eponline\":{\"permissions\":[],\"addRemoveGroups\":false},\"epcampus\":{\"permissions\":[],\"addRemoveGroups\":false},\"epinstructor\":{\"permissions\":[],\"addRemoveGroups\":false},\"confirmed\":{\"permissions\":[\"reupload\",\"upload\",\"move\",\"collectionsaveasuserpage\",\"collectionsaveascommunitypage\",\"autoconfirmed\",\"editsemiprotected\",\"movestable\",\"autoreview\",\"skipcaptcha\",\"abusefilter-log-detail\"],\"addRemoveGroups\":false},\"copyviobot\":{\"permissions\":[\"pagetriage-copyvio\"],\"addRemoveGroups\":false}}",
  "mmUserRights": "[\"extendedconfirmed\",\"suppressredirect\",\"move-subpages\",\"move\",\"patrol\",\"review\",\"autoreview\",\"autoconfirmed\",\"editsemiprotected\",\"rollback\",\"templateeditor\",\"tboverride\",\"createaccount\",\"read\",\"edit\",\"createtalk\",\"writeapi\",\"viewmywatchlist\",\"editmywatchlist\",\"viewmyprivateinfo\",\"editmyprivateinfo\",\"editmyoptions\",\"abusefilter-log-detail\",\"centralauth-merge\",\"abusefilter-view\",\"abusefilter-log\",\"vipsscaler-test\",\"collectionsaveasuserpage\",\"reupload-own\",\"move-rootuserpages\",\"move-categorypages\",\"createpage\",\"minoredit\",\"editmyusercss\",\"editmyuserjson\",\"editmyuserjs\",\"purge\",\"sendemail\",\"applychangetags\",\"spamblacklistlog\",\"mwoauthmanagemygrants\",\"reupload\",\"upload\",\"collectionsaveascommunitypage\",\"movestable\",\"skipcaptcha\",\"transcode-reset\"]",
  "mmv.hasOpenedMetadata": "*",
  "mobile-betaoptin-token": "*",
  "mw-mobile-sidebar-state": "hidden",
  "mw-revslider-hide-help-dialogue": "*",
  "mwedit-state-wikibaseEntityUsage": "*",
  "uls-previous-language-autonym": "English",
  "uls-previous-language-code": "en",
  "ve-beta-welcome-dialog": "*",
  "wmE-sS--sessionId": "rejected",
  "wmE-sS--sessionIdEndTime": "*"
}"
Imarlier assigned this task to Krinkle.Sep 24 2018, 8:11 PM
Imarlier moved this task from Inbox to Doing on the Performance-Team board.
MMiller_WMF moved this task from To Triage to External on the Growth-Team board.Sep 24 2018, 8:15 PM

@JFG Thanks. I've looked into a few more possibilities:

  • ULS (UniversalLanguageSelector): There are ULS modules loaded in your report and ULS has logic that can be influenced by local data. But, your ULS settings do not trigger a memory leak as far as I can tell. I've also looked over the ULS code and did not code that I believe would execute repeatedly or otherwise cause growth in memory use.
  • CentralNotice: CentralNotice loaded for you and you've got local data about past banner impressions. The CentralNotice extension has logic that runs in the background to proactively remove data that is no longer needed (ext.centralNotice.kvStoreMaintenance). This code does execute repeatedly over a long period of time, and it interacts with localStorage, but.. as far as I can tell, it only reduces memory.

I can look at some more modules, but wanted to first look at the Memory Profile you submitted. From a quick glance, I observed four things:

  1. The Memory Profile capture spans 32 seconds of real time. Memory use was initially at 12.9 MiB and remained mostly static, except for a single point in the middle where it increased by 300K to 13.2 MiB, then it remains constant again at that level. This suggests to me that either the culprit was not seen during this profile capture, or.. the one increasing point must be the culprit.
  2. I saw a function called rtcScript() that runs very frequently during the profile. I could not find any code in the Wikimedia source trees by this name, neither on English Wikipedia's user scripts or gadgets. I suspect this function may be coming from a browser extension you have installed. It runs every 0.1s, which makes it hard to rule out as possible cause. Given it runs every 0.1s, it naturally does intersect with the moment where memory use increased. On the other hand, it also runs throughout the rest of the profile capture without memory increase.
  3. As you mentioned, there are a few mouseover/mouseout events. Most of these don't cause any problem, so I ignored them. Each event triggered a short execution of logic from jQuery, but jQuery discards the event as not matching any of the specified listeners, and then jQuery cleans up after itself.
  4. The rest of the profile capture is almost entirely clean. No code runs, no change in memory. Except for the Page Previews that appear when a link is moused-over in the article. And it seems after each of those, the memory does increase without going back down.

Next, I'll in more detail at the activity your Profile capture that relates to the Page Previews code (aka "extensions/Popups"):

Krinkle added a comment.EditedSep 25 2018, 12:23 AM

Here is a summary of the activity in Performance capture submitted @JFG, reduced to things relating to Popups.

Screenshot of JFG's profile (zoomed out):

Time and durationActivityJS Heap
at 16,390msEvent (mouseover)+9 KB
+14ms jQuery.event.dispatch, Sizzle selector matcher, end.From 13576008 B to 13585248 B.
at 16,410msEvent (mouseout)+7 KB
+3ms jQuery.event.dispatch, Sizzle selector matcher, end.From 13585248 B to 13592328 B.
at 17,325msEvent (mouseover)+71 KB
+10ms jQuery.event.dispatch, Sizzle selector matcher, invoke Popups callback.From 13592328 B to 13663752 B
Popups: (things that stand out) Create mw.Uri, create mw.Title, create $.Deferred, install Timer, call mw.user.generateRandomSessionId, call $.fn.hide.
at 17,335msEvent (mouseout)+67 KB
+44ms jQuery.event.dispatch, Sizzle selector matcher, invoke Popups callback.From 13663752 B to 13730488 B
Popups: (things that stand out) call promise.abort (deferred.reject), 2x create $.Deferred via promise.then and promise.catch(), 2x install Timer (for deferred.then)
at 17,409msTimer Fired+2.7 KB
+0.1ms Popups callback: call clearTimeout, remove Timer.From 13730488 B to 13733240 B
at 17,410msTimer Fired+5.2 KB
+0.15ms jQuery internals (Deferred: mightThrow, Thrower)From 13733240 B to 13735776 B
at 17,679msTimer Fired+ 4 KB
+0.7ms Popups callback: call deferred.then, end.From 13735776 B to 13740344 B
at 16,681msTimer Fired+ 46 KB
+6ms $.Deferred.dispatch, invoke Popups callbackFrom 13740344 B to 13787184 B
Popups callback: call $.extend, call mw.track, handleTrackedEvent, call mw.loader.using, call mw.eventLog.logEvent, send Beacon, send Request, create $.Deferred

At every single step, both jQuery/Popups appear to keep adding to the amount of persistently held memory. It's therefore quite likely that something in this code is keeping references to objects that should no longer be relevant, and should be released, but aren't.

The only data relevant to this code that I know needs to be retained, is the internal memory queue used by mw.track(), which i a fairly insignificant part of the story above. About ~0.9 KB is allocated by mw.track, once, in the last row of the above table.

The question is, where has the 211 KB gone?

Krinkle renamed this task from Idle Wikipedia pages cause a memory leak to Memory leak from Popups on Wikipedia pages.Sep 25 2018, 12:24 AM
Krinkle triaged this task as Normal priority.Sep 25 2018, 12:34 AM
JFG added a comment.Sep 25 2018, 6:40 AM

Thanks for your detective work, @Krinkle. FYI I have taken longer samples but I could not see much relevant activity in there. Given the clear-cut difference in modules when a page leaks and does not leak, I would suggest to focus our investigations on the a11y service (accessibility). Does this belong to core mediawiki code? Is there a way to disable it in my account settings? Can I produce a memory trace that would specifically focus on events linked to this module?

JFG added a comment.Sep 25 2018, 6:42 AM

Also, I don't think page previews are very relevant to the central case. I may have triggered that code by mousing over the page in this particular session, but most of my tests showing large long-term leaks happened with the window minimized, thus without any user interaction at all.

JFG added a comment.Sep 25 2018, 12:21 PM

Latest experiment: when I check "Disable access keys" in preferences, the ext.cite.a11y module is not loaded (neither are its friends jquery.tablesorter and jquery.makeCollapsible), but the page still leaks.

Thanks, by process of elimination that confirms the hypothesis of the leak coming from Page Previews. In addition, the memory profile has already confirmed that its event handlers are leaking memory.

However, it is possible you are experiencing a second leak as well. To know for sure, I'd need another memory profile where other increases are found. May I ask how you found the memory increase originally? What tool or method did you use to find that it came from Wikipedia?

JFG added a comment.Sep 25 2018, 7:48 PM

May I ask how you found the memory increase originally? What tool or method did you use to find that it came from Wikipedia?

I have been watching memory usage in the Chrome task manager.

phuedx added a subscriber: phuedx.Sep 26 2018, 8:26 AM
Krinkle moved this task from Doing to Blocked on the Performance-Team board.Sep 26 2018, 11:41 PM
JFG added a comment.Oct 1 2018, 10:23 AM

@Krinkle: anything I can do to help track down the issue?

JFG added a comment.Oct 7 2018, 2:09 PM

Update: I have disabled the new page reviewer user permission, and the memory still leaks. Let me know if you need any new reports.

Pinging @Prtksxna, who is listed at https://www.mediawiki.org/wiki/Developers/Maintainers as the maintainer for this tool.

Pinging @Prtksxna, who is listed at https://www.mediawiki.org/wiki/Developers/Maintainers as the maintainer for this tool.

Since the Reading team took over the extension I have not been maintaining it. Sorry I didn't update that page sooner.

Krinkle removed Krinkle as the assignee of this task.Oct 14 2018, 9:34 PM
Krinkle edited projects, added Performance-Team (Radar); removed Performance-Team.
Krinkle added a subscriber: Jdlrobson.

@Niedzielski @Jdlrobson Moving this to our radar as the initial investigation has been completed, and a probable source has been identified (in this case, the Page Previews feature). Let me know if and when further help is needed.

Krinkle moved this task from Watching to Backlog on the Readers-Web-Backlog (Tracking) board.
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.

Task manager observations for my own consideration later.

When I only leave a background Wikipedia tab open, the size does not increase much:

When I actively browse other Wikipedia pages in the foreground tab, the background tab's size grows quickly:

The leak occurs whether the page previews UI is enabled or disabled.

The above occurs on mobile which does not have Popups.

@Krinkle I'm having a lot of difficulty getting up to speed with what the exact problem is here. Can you please update the description with a summary of what is known and what is not known?

Do we know exactly where the memory leak in Popups is, or do we only know that it relates to Popups? Is there more to investigate here?

Krinkle added a comment.EditedOct 16 2018, 6:27 PM

See T205127#4613124 and T205127#4613193 for a recently written summary. What I know is that users of several different communities have reported through various channels over the past 2-3 months about memory leaks, after several years of having virtually heard nothing about the subject. They could be unrelated reports, but so far they had in common:

  • Wikipedia (English and German).
  • Multiple browsers (Chrome, Opera, possibly others).
  • Desktop (any skin).
  • Logged-in.

This task in particular shows:

  • There may be multiple sources of memory leaks. As such, let's focus this task only on the issue reported by @JFG.
  • The report from @JFG via @Whatamidoing-WMF includes various bits of information as well as a Chrome trace log.
  • This trace log shows direct correlation on an otherwise idle page that every mouse event that triggered significant code in Popups, also had a subsequent increase in memory use without a decrease in memory at any point. Specific function names, context and reproduction steps are found in T205127#4613193.

While there maybe other leaks in other places, I suggest to create separate tasks about each finding so that we can triage, understand and resolve each one individually.

JFG added a comment.EditedOct 17 2018, 10:54 AM

Thanks for the new summary. I'll add that we have eliminated a bunch of other possible causes, by removing browser extensions, user scripts and gadgets. I started noticing those leaks in early September, as they became monumental (idle pages consuming more than 1 GB of memory before I kill them manually). I thought they might be related to the "new page reviewer" user right that I had recently acquired, but I have since opted out of this right and the problem persists.

I see the same behavior that was noted by @Niedzielski above: background pages in separate processes gobble up memory when I'm active on foreground pages. Also, bringing back a background page to the foreground and editing it clears up the memory outgrowth instantly. However, I have only noticed the issue on the English Wikipedia: German, French, Russian, Japanese versions and Wikidata do not seem to leak in my case. I have not yet tried actively working on foreground pages of those wikis while an idle one is in the background. Will test that asap.

I am doubtful that the popups issue explains the largest leaks, but of course we have identified a leak there, that should be investigated. I'm happy to run any tests required to help isolate the issue further.

JFG added a comment.Oct 17 2018, 1:10 PM

Correction, and maybe an important clue: when a page in the background has been leaking memory, merely bringing it to the foreground and scrolling around in it is sufficient to reduce the memory footprint back to normal. No need to edit the page, just view it. With more careful experimenting, it appears that the memory is released as soon as a link mouseover event is processed and a thumbnail preview displayed. When I just open the relevant tab and carefully scroll from the sides without mousing over any text, the memory usage remains high. This behavior apparently establishes a strong connection between the popups system and the memory leak, and it's easy to reproduce. I tried to take a sample execution trace from the console to capture the release of memory, but that did not work because the leaked memory also gets released as soon as I open the dev tools panel. Hope this helps. Let me know what I can attempt next.

With more careful experimenting, it appears that the memory is released as soon as a link mouseover event is processed and a thumbnail preview displayed. When I just open the relevant tab and carefully scroll from the sides without mousing over any text, the memory usage remains high.

Well spotted @JFG! I see the same thing on Firefox Nightly 64.

JFG added a comment.Oct 17 2018, 1:19 PM

I just tested the working-in-foreground-while-idle-page-in-background case on the German Wikipedia. Confirmed that:

  1. the background page does not leak as long as it's the only open page on dewiki, it can be stable for hours
  2. even a small amount of work on another dewiki page (open article, click edit, click preview) triggers 30-40 MB of leak in the background page
  3. bringing the background page back to the foreground by itself causes no change, scrolling carefully is also fine
  4. triggering a popup with a mouseover on any link resets the memory usage to its baseline value from step 1

Please disregard my previous comments on the phenomenon being limited to enwiki: that was because I had not tested background pages on other wikis while working on foreground pages of the same wiki.

JFG added a comment.Oct 17 2018, 1:32 PM

Another test whereby the leak is easily triggered with minimal steps:

  1. open a first page, read it, scroll around, let it come to a stable memory footprint (as monitored in the Chrome task manager)
  2. open another page in a different tab, make sure it has a different process ID, just let it render without moving the mouse into it
  3. watch the memory footprint of both pages and do a single mouseover on a link within the foreground page
  4. the background page consumes a few extra megabytes of memory!
  5. keep reading the foreground page; very soon the background page consumes more memory than the foreground page…
JFG added a comment.EditedOct 17 2018, 1:41 PM

New test: the leak-in-background-page phenomenon also occurs when logged out.

The leak occurs whether the page previews UI is enabled or disabled.

I can confirm this as well.

@JFG To confirm - The leak remains when bringing focus to the background window and making it the foreground window and scrolling. It stops when you mouse over a link, even when the mousing over doesn't do anything (when previews is disabled).

This would lead me to one of three possible causes:

  1. There is a leak in the Chrome browser outside our control where somehow a native thread is building up memory for unknown reasons an releasing it on the first I/O event received over a hyperlink.
  2. There is a leak in the Page Previews code, either the one we previously identified or a second one, in generic Popups code that fires even when the feature is disabled. We could eliminate this if we can confirm that none of its code is loaded when the feature is disabled, and that @JFG disabled it via Special:Preferences, cleared the browser cache (only cache, no need to clear cookies), and reproduce the issue on a page view.
  3. There is a leak in some other JavaScript code shipped by MediaWiki core, an extension, or a default-enabled gadget on en.wikipedia.org that somehow relates to mouse events.
JFG added a comment.EditedOct 17 2018, 8:56 PM

@Krinkle: I tried the process you describe on the German Wikipedia, to avoid being polluted by my daily work on the English site. Here's what I have done:

  • disabled page preview in preferences
  • fully purged the browser cache
  • loaded a dewiki article in a new tab, making sure it had its own process ID (background page)
  • opened another dewiki article in another tab, making sure it had a separate process ID (foreground page)
  • browsed around in the foreground page, while checking memory usage on both pages
  • noticed memory usage of the background page increasing when left untouched, while browsing in the foreground page
  • saw an 8MB leap of memory usage on the background page when I hovered over a link in the foreground page. The link hovering just displayed a tooltip with the name of the target article, "Wirtschaftswunder".
  • saw a 12 MB leap on the background page when I opened another article in the foreground page, while it was being rendered
  • every new page rendered in the foreground process triggered a 7-8 MB memory increase in the background page's process
  • noticed some tiny amount of CPU usage in the background process every time I hovered over a link in the foreground page (about 0.1% CPU visible during one or two frames of the 1-second refresh rate of the task manager). That usually did not trigger a memory increase.
  • noticed some more significant CPU usage in the background process every time the foreground process finishes loading and rendering a new page (typically 2.7–3.5% during one frame), happening in sync with the memory increase of 7-8 MB. This usually happens with a delay of a couple seconds after the new page is loaded.
  • opening a picture within an article of the foreground process also triggers a slightly-delayed 7-8 MB memory increase and 1-second 3% CPU usage in the background process
  • browsing through several images in the foreground page once in gallery mode does not change the memory allocation in the background process, even on articles with many pictures.
  • throughout all these actions, memory usage of the active foreground process goes up and down, and remains far under what happens to the background process.
  • during all the activities in the foreground process, I have not seen any network usage triggered in the background process
  • when switching to the background page, memory usage remains high, it can even increase. Scrolling does not change it.
  • hovering over links in the background page (now foreground) does not reduce the memory footprint (that differs from the case when popups were active -- the popup display was clearing the extra memory), and it does consume some CPU (5-6% during 2-3 seconds). Again, only the tooltip with target article name gets displayed.
  • after clicking a link to load another article, memory is purged and returns to a normal level.

Memory bugs are always ''interesting''…

@JFG in Google Chrome, if you open the Task Manager, right click on the columns at the top and add "JavaScript Memory", do you see that value increasing or just memory footprint?

I only see memory footprint increasing, not JavaScript memory, which indicates possibly detached DOM tree memory leaks. Unfortunately I didn't spot anything obvious from looking at heap snapshots.

JFG added a comment.EditedOct 18 2018, 9:54 PM

@kostajh: The JavaScript memory is stable in the background page.

JFG added a comment.Oct 26 2018, 9:17 AM

@kostajh, @Krinkle: bumping thread.

Any idea where to send this report now that triggering conditions are narrowed down, and it's been well reproduced? A gigabytes-sized memory leak is a rather annoying state of affairs for one of the world's top sites. It especially harms active editors, and we need more of them to maintain quality and coverage. Hope you can identify who would be the best people to address this.

That would be my team. I need to dig through this thread and understand the problem fully. We are coming up to a big release so i haven't found the time yet.

Thank you for all the input and investigation.

I'm at an offsite starting today and will bring it up.

Tbayer added a subscriber: Tbayer.Nov 2 2018, 11:53 PM

See T205127#4613124 and T205127#4613193 for a recently written summary. What I know is that users of several different communities have reported through various channels over the past 2-3 months about memory leaks, after several years of having virtually heard nothing about the subject. They could be unrelated reports, but so far they had in common:

For the record, here's the link to one of the reports @Krinkle referred to:
https://de.wikipedia.org/wiki/Wikipedia:Fragen_zur_Wikipedia/Archiv/2018/Woche_37#WP_als_Speicherfresser (Opera 55.0.2994.56 under Ubuntu 16.04)

I also seem to recall encountering such an issue myself repeatedly in the last 2 months or so, with a single tab growing toward 2GB, in Chromium under Ubuntu, logged in on desktop.

First things first we'll get one of the devs who know's the popups code to understand what's going on here in a more isolated environment... T208634

JFG added a comment.Nov 3 2018, 9:21 AM

Sure. But please note the issue may not be related to popups.

@JFG for now, I tried to isolate issues that only refer to Popups - I'm trying to learn how to properly use the memory profiling tools. I'll keep hunting for memory leaks in whole WMF code.

JFG added a comment.Fri, Dec 7, 1:04 AM

@pmiazga: Your work on phab:T208634 looks promising. Let me know whenever there's a potential fix to test.