Page MenuHomePhabricator

ResourceLoader urls with certain version serving different content on beta cluster
Closed, ResolvedPublic

Description

Note grepping for toast.showOnPageReload you'll find it is present in http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=mobile.editor.api%2Ccommon%2Coverlay&skin=minerva&version=da39a3ee5e6b&* but mobile-pending-toast isn't

The inverse is true for http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=mediawiki.confirmCloseWindow%7Cmobile.abusefilter%2CmicroAutoSize%7Cmobile.editor.api%2Ccommon%2Coverlay%7Coojs-ui%7Coojs-ui.styles%7Coojs-ui.styles.icons%2Cindicators%2Ctextures&skin=minerva&version=da39a3ee5e6b&* despite version being the same.

Basically different code is being loaded for 2 urls for unknown reasons. This is breaking mobile browser tests by serving incompatible code.

Maybe caused by the fix for T94074 ? I don't understand enough of this stuff to debug any more but please let me know how I can resolve this.

Event Timeline

Jdlrobson claimed this task.
Jdlrobson raised the priority of this task from to High.
Jdlrobson updated the task description. (Show Details)
Jdlrobson added subscribers: Krinkle, Aklapper, Jdlrobson.
greg renamed this task from ResourceLoader urls with certain version serving different content on beta labs to ResourceLoader urls with certain version serving different content on beta cluster.May 27 2015, 7:23 PM
greg set Security to None.

On beta the the bits cache is not invalidated and I have no idea how it is done in production. See also T65034: Caching makes it impossible to test JS changes when logged out or dupe T90983.

If that is the issue (stalled cache), you might want to merge it to T65034.

To debug cache I usually do: curl --verbose http://somewhere > /dev/null and look at the headers. For the two URLs above I got:

http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=mobile.editor.api%2Ccommon%2Coverlay&skin=minerva&version=da39a3ee5e6b&*

< Expires: Fri, 19 Jun 2015 12:51:52 GMT
< ETag: W/"pKVIxHZ8"
< Age: 721764
< X-Cache: deployment-cache-text02 miss (0), deployment-cache-text02 frontend hit (163)

So got it 163 times, will expire on June 19th and has been in cache for 8+ days.

The second http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=mobile.editor.api%2Ccommon%2Coverlay&skin=minerva&version=da39a3ee5e6b&* has:

< Expires: Fri, 26 Jun 2015 18:40:32 GMT
< ETag: W/"7+0Ipx0d"
< Age: 96063
< X-Cache: deployment-cache-text02 miss (0), deployment-cache-text02 frontend hit (5)

Feel free to copy paste on T65034.

Again, I have absolutely no clue how the cache is invalidated on bits in prod context. Maybe it is solely based on ETag but no idea how it is generated either.

More header fun

Here is the backend from which Varnish caches, the source

❯ curl -I -H 'host: en.wikipedia.beta.wmflabs.org' 'http://10.68.16.127/w/load.php?debug=false&lang=en&modules=mobile.editor.api%2Ccommon%2Coverlay&skin=minerva&version=da39a3ee5e6b&*'
HTTP/1.1 200 OK
Date: Thu, 28 May 2015 21:35:47 GMT
Server: Apache
X-Powered-By: HHVM/3.6.1
X-Content-Type-Options: nosniff
Cache-control: public, max-age=2592000, s-maxage=2592000
Vary: Accept-Encoding
Expires: Sat, 27 Jun 2015 21:35:47 GMT
ETag: W/"m+kMoFiY"
Content-Type: text/javascript; charset=utf-8

Here is the backend varnish server, from which the frontend varnish server caches

Noteworthy here is the ETag is different than the source

❯ curl -I -H 'host: en.wikipedia.beta.wmflabs.org' 'http://10.68.16.16:3128/w/load.php?debug=false&lang=en&modules=mobile.editor.api%2Ccommon%2Coverlay&skin=minerva&vers
ion=da39a3ee5e6b&*'
HTTP/1.1 200 OK
Server: Apache
X-Powered-By: HHVM/3.6.1
X-Content-Type-Options: nosniff
Cache-control: public, max-age=2592000, s-maxage=2592000
Vary: Accept-Encoding
Expires: Fri, 26 Jun 2015 18:40:32 GMT
ETag: W/"7+0Ipx0d"
Content-Type: text/javascript; charset=utf-8
Date: Thu, 28 May 2015 21:40:58 GMT
X-Varnish: 854690096 853638629
Age: 97226
Via: 1.1 varnish
Connection: keep-alive
X-Cache: deployment-cache-text02 hit (1)

Frontend text cache varnish server

has setcookie headers, but largely the same as the backend varnish server

❯ curl -I -H 'host: en.wikipedia.beta.wmflabs.org' 'http://10.68.16.16/w/load.php?debug=false&lang=en&modules=mobile.editor.api%2Ccommon%2Coverlay&skin=minerva&version=d
a39a3ee5e6b&*'
HTTP/1.1 200 OK
Server: Apache
X-Powered-By: HHVM/3.6.1
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
Expires: Fri, 26 Jun 2015 18:40:32 GMT
ETag: W/"7+0Ipx0d"
Content-Type: text/javascript; charset=utf-8
X-Varnish: 853638629, 1186813790 1186669704
Via: 1.1 varnish, 1.1 varnish
Date: Thu, 28 May 2015 21:35:15 GMT
Age: 96882
Connection: keep-alive
X-Cache: deployment-cache-text02 miss (0), deployment-cache-text02 frontend hit (6)
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Set-Cookie: GeoIP=::::v4; Path=/; Domain=.wmflabs.org
Set-Cookie: WMF-Last-Access=28-May-2015;Path=/;HttpOnly;Expires=Mon, 29 Jun 2015 12:00:00 GMT
BBlack added a subscriber: BBlack.May 28 2015, 11:36 PM

This is all related to the same basic issue in production: T99096

Note grepping for toast.showOnPageReload you'll find it is present in http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=mobile.editor.api%2Ccommon%2Coverlay&skin=minerva&version=da39a3ee5e6b&* but mobile-pending-toast isn't
The inverse is true for http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=mediawiki.confirmCloseWindow%7Cmobile.abusefilter%2CmicroAutoSize%7Cmobile.editor.api%2Ccommon%2Coverlay%7Coojs-ui%7Coojs-ui.styles%7Coojs-ui.styles.icons%2Cindicators%2Ctextures&skin=minerva&version=da39a3ee5e6b&*

Both of these urls are affected by a regression that was fixed last week. The regression was causing the cache to properly update once after T94074, but then never again. This bug was fixed in c5d6521d8f.

Krinkle closed this task as Resolved.Jun 3 2015, 12:22 AM

This was caused due to a temporary regression while merging T94074. This has been fixed since and is not related in any way to T65034 or T99096.