Change Details

`E3` deployed several extensions with changed modules today. GettingStarted had a new RL module 'ext.gettingstarted.openTask' for a new JS file resources/ext.gettingstarted.openTask.js. scap finished 4:19PDT [00:19:44] 50 minutes after scap finished I visited a page that should have had the new module, but didn't. mw.loader.getState( 'resources/ext.gettingstarted' ) reported "missing". Shift-reload of the page didn't help; requesting just the bits URL that included this module [1] also reported it "missing"[2]. Modifying the version in the query string to a later time or requesting just the ext.gettingstarted.openTask module worked fine. My hypothesis: during scap, some user requested a page that ran the new PHP code that adds the new module, so her browser made the request to bits.wikimedia.org for modules including the new module. At that time the bits server didn't yet have the JS file, so the server response reports "missing". But then RL happily caches that response for X minutes, where X is a long time, well over 50 minutes. The missing module continued for about 10 more minutes, until the same URL worked.[3] The working request reported a modified time of Thu 07 Mar 2013 04:07:51 PM PST (I think 00:07:51 UTC, which matches the version= on the end of the bits URL. The a-m-c times of the ext.openTask.js file according to stat(1) are Access: 2013-03-08 00:08:12.857544926 +0000 Modify: 2013-03-07 21:49:43.000000000 +0000 Change: 2013-03-08 00:08:12.857544926 +0000 the same as the other files in the module that existed before this deploy. Is the workaround to avoid these damaging situations to touch every JS and CSS file that one deploys before running scap? This might be a dupe of T39812 , "Module cache should be invalidated (newer timestamp) when lesser old items are removed or added (scripts, style, messages)" [1]: The request with missing module: https://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=ext.Experiments.lib|ext.UserBuckets%2CeventLogging%2CmarkAsHelpful%2CpostEdit|ext.articleFeedbackv5.startup|ext.gadget.DRN-wizard%2CHotCat%2CReferenceTooltips%2Ccharinsert%2CmySandbox%2Cteahouse|ext.gettingstarted.openTask|jquery.articleFeedbackv5.verify|jquery.autoEllipsis%2CcheckboxShiftClick%2CclickTracking%2CdelayedBind%2Chidpi%2ChighlightText%2Cjson%2CmakeCollapsible%2Cmw-jump%2Cplaceholder%2Csuggestions%2CtabIndex|mediawiki.Title%2CUri%2Capi%2Chidpi%2CsearchSuggest%2Cuser|mediawiki.api.watch|mediawiki.page.ready|mediawiki.page.watch.ajax|mobile.desktop|mw.MwEmbedSupport.style|mw.PopUpMediaTransform|schema.GettingStarted&skin=vector&version=20130308T000751Z&* [2]: The end of the (jsbeautify'd) response for this URL was: ``` }, {}, {}); mw.loader.state({ "ext.gettingstarted.openTask": "missing" }); /* cache key: enwiki:resourceloader:filter:minify-js:7:3e6c3b5e8a0b731930af1060b6ab62b6 */ ``` [3]: The identical URL request when it starting working returned a last line of: ``` /* cache key: enwiki:resourceloader:filter:minify-js:7:03dcfbab169aa00f66478e647270d340 */ ```

@spage wrote: > `E3` deployed several extensions with changed modules today. GettingStarted had a new RL module 'ext.gettingstarted.openTask' for a new JS file resources/ext.gettingstarted.openTask.js. scap finished 4:19PDT [00:19:44] > 50 minutes after scap finished I visited a page that should have had the new code, but didn't. > > My hypothesis: during scap, some user requested a page that ran the new PHP code that adds the new module, so her browser made the request to bits.wikimedia.org for modules including the new module. At that time the bits server didn't yet have the JS file, so the server response reports "missing". But then RL happily caches that response for X minutes, where X is a long time, well over 50 minutes. > > This might be a dupe of T39812 , "Module cache should be invalidated (newer timestamp) when lesser old items are removed or added (scripts, style, messages)" It turned out the original report above was indeed T39812. However there is a larger issue here. During deployment, any change that introduces use of a new URL that is deterministically versioned and has large max-age, has a risk of indefinitely polluting the cache. The time line is as follows: * Start deployment (scap, sync-file, sync-dir). * Server A receives new code deployment. * Client 1 request a page. * Server A responds to client 1 with the page, build with the new code. * Client 1 receives page from server A and requests any secondary resources (e.g. referenced script, style or image). – This resource has URIs like `/static/file-name?{hash}` or `/load.php?module=name&{hash}`. * Server B responds to client 1 and provides the requested resource based on its name. * Server B receives new code deployment. Given we have Varnish in front of web servers, this means the outdated resource from server B is now cached at the url containing the new hash. As such, even once the deployment is over, it will continue to persist in cache indefinitely. When adding new static files, server B would respond with 404 and it will repair itself eventually when the file exists. However for an update to static files or a change anywhere in JS/CSs modules, this can cause the Vanish cache to effectively **renew old content under a new version URIs**.

`E3` deployed several extensions with changed modules today. GettingStarted had a new RL module 'ext.gettingstarted.openTask' for a new JS file resources/ext.gettingstarted.openTask.js. scap finished 4:19PDT [00:19:44]@spage wrote: 50 minutes after scap finished I visited a page that should have had the new module, but didn't. mw.loader.getState( 'resources/ext.gettingstarted' ) reported "missing". Shift-reload of the page didn't help;> `E3` deployed several extensions with changed modules today. requesting just the bits URL that included this module [1] also reported it "missing"[2]. Modifying the version in the query string to a later time or requesting just the GettingStarted had a new RL module 'ext.gettingstarted.openTask module worked fine. My hypothesis: during scap, some user requested a page that ran the new PHP code that adds the new module, so her browser made the request to bits.wikimedia.org for modules including the new module. At that time the bits server didn't yet have the JS file, so the server response reports "missing". But then RL happily caches that response for X minutes, where X is a long time, well over 50 minutes. The missing module continued for about 10 more minutes, until the same URL worked.[3] The working request reported a modified time of Thu 07 Mar 2013 04:07:51 PM PST (I think 00:07:51 UTC,' for a new JS file resources/ext.gettingstarted.openTask.js. which matches the version= on the end of the bits URL. The a-m-c times of the ext.openTask.js file according to stat(1) arescap finished 4:19PDT [00:19:44] Access: 2013-03-08 00:08:12.857544926 +0000> 50 minutes after scap finished I visited a page that should have had the new code, but didn't. > > My hypothesis: during scap, some user requested a page that ran the new PHP code that adds the new module, so her browser made the request to bits.wikimedia.org for modules including the new module. At that time the bits server didn't yet have the JS file, so the server response reports "missing". Modify: 2013-03-07 21:49:43.000000000 +0000But then RL happily caches that response for X minutes, where X is a long time, well over 50 minutes. Change: 2013-03-08 00:08:12.857544926 +0000 the same as the other files in the module that existed before this deploy. Is the workaround to avoid these damaging situations to touch every JS and CSS file that one deploys before running scap?> > This might be a dupe of T39812 , "Module cache should be invalidated (newer timestamp) when lesser old items are removed or added (scripts, style, messages)" This might be a dupe of T39812 , "Module cache should be invalidated (newer timestamp) when lesser old items are removed or added (scripts, style,It turned out the original report above was indeed T39812. messages)"However there is a larger issue here. [1]: The request with missing module:During deployment, any change that introduces use of a new URL that is deterministically versioned and has large max-age, has a risk of indefinitely polluting the cache. https://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=ext.Experiments.lib|ext.UserBuckets%2CeventLogging%2CmarkAsHelpful%2CpostEdit|ext.articleFeedbackv5.startup|ext.gadget.DRN-wizard%2CHotCat%2CReferenceTooltips%2Ccharinsert%2CmySandbox%2Cteahouse|ext.gettingstarted.openTask|jquery.articleFeedbackv5.verify|jquery.autoEllipsis%2CcheckboxShiftClick%2CclickTracking%2CdelayedBind%2Chidpi%2ChighlightText%2Cjson%2CmakeCollapsible%2Cmw-jump%2Cplaceholder%2Csuggestions%2CtabIndex|mediawiki.Title%2CUri%2Capi%2Chidpi%2CsearchSuggest%2Cuser|mediawiki.api.watch|mediawiki.page.ready|mediawiki.page.watch.ajax|mobile.desktop|mw.MwEmbedSupport.style|mw.PopUpMediaTransform|schema.GettingStarted&skin=vector&version=20130308T000751Z&*The time line is as follows: [2]: The end of the (jsbeautify'd) response for this URL was: ```* Start deployment (scap, sync-file, sync-dir). }, {}, {});* Server A receives new code deployment. mw.loader.state({* Client 1 request a page. "ext.gettingstarted.openTask": "missing"* Server A responds to client 1 with the page, build with the new code. });* Client 1 receives page from server A and requests any secondary resources (e.g. referenced script, style or image). – This resource has URIs like `/static/file-name?{hash}` or `/load.php?module=name&{hash}`. /* cache key: enwiki:resourceloader:filter:minify-js:7:3e6c3b5e8a0b731930af1060b6ab62b6 */* Server B responds to client 1 and provides the requested resource based on its name. ```* Server B receives new code deployment. [3]: The identical URL request when it starting working returned a last line of: ``` /* cache key: enwiki:resourceloader:filter:minify-js:7:03dcfbab169aa00f66478e647270d340 */Given we have Varnish in front of web servers, this means the outdated resource from server B is now cached at the url containing the new hash. As such, even once the deployment is over, it will continue to persist in cache indefinitely. ```When adding new static files, server B would respond with 404 and it will repair itself eventually when the file exists. However for an update to static files or a change anywhere in JS/CSs modules, this can cause the Vanish cache to effectively **renew old content under a new version URIs**.