Page MenuHomePhabricator

[Research] Investigate how long startup module is cached by clients
Closed, ResolvedPublic

Description

Per @Catrope, we have reason to believe clients (varnish, and/or browsers) may be caching the startup module longer than they should.

When they deployed a change to a configuration variable embedded in mw.config, it took several hours of slow ramp up for all clients to be running with the new values. The config change in question changed how VisualEditor loaded the HTML data model (e.g. from RESTBase instead of Parsoid).

This was based how the traffic volume changed for those two endpoints.

I propose we run a campaign in WikimediaEvents/EventLogging (or perhaps simpler using statsv) that essentially just reflects back the value of a dummy config var. And then after a week or so we change its value.

If we determine that this is indeed problematic, one potential remedy would be adding must-revalidate to our Cache-Control header for short-lived load.php responses.

Event Timeline

Krinkle created this task.Jul 9 2015, 12:00 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.
Krinkle added subscribers: Gilles, ori, Krinkle and 2 others.

Our actual observations were based on sending timing data to EventLogging under different names depending on which HTML backend was in use, so it was actually equivalent to your proposal of echoing back the value of a config var (except that in our case the volume was weighted by how many times the affected users used VE).

In theory, the startup module should never be more than about 10 minutes old (5 minutes in Varnish cache, 5 minutes in the client cache). This number might be higher if there are other caches between us and the client (add 5 minutes for each cache), but in an HTTPS-only world that seems unlikely (our VE observation was 2-3 months ago, before the switch to HTTPS-only). Furthermore, because there are only three backend Varnishes that can cache the startup module response (one in each of eqiad, ulsfo and esams), the server-side component of this delay should be sudden for entire continents at once (whereas the client-side component can be gradual, once a Varnish cache has turned over). So you'd expect to see three jumps in the first 5 minutes as each of the Varnishes turns over (these jumps may coincide), each followed by a gradual turnover of client caches spread over 5 minutes. After 10 minutes, the proportion of clients with the old startup module should be negligible.

In practice, we observed EventLogging traffic gradually migrating from the old name to the new name with most of the change happening in the first 15-30 minutes, and a small portion sticking around for hours. Grafana doesn't appear to have these graphs any more, so it would be nice to repeat this experiment in a non-accidental fashion and document the results clearly.

Krinkle claimed this task.Jul 27 2015, 9:02 PM
Krinkle triaged this task as Normal priority.
Krinkle moved this task from Inbox to Doing on the Performance-Team board.
Krinkle moved this task from Backlog to Assigned on the MediaWiki-ResourceLoader board.

Change 227358 had a related patch set uploaded (by Krinkle):
Add temporary rl-test.php entry point

https://gerrit.wikimedia.org/r/227358

Change 227396 had a related patch set uploaded (by Catrope):
ResourceLoader: Add must-revalidate to Cache-Control

https://gerrit.wikimedia.org/r/227396

Change 227396 merged by jenkins-bot:
resourceloader: Add must-revalidate to Cache-Control

https://gerrit.wikimedia.org/r/227396

Change 227358 merged by jenkins-bot:
Add temporary rl-test.php entry point

https://gerrit.wikimedia.org/r/227358

Change 228124 had a related patch set uploaded (by Krinkle):
rl-test: Add instrumentation for User-Agent and Remote IP

https://gerrit.wikimedia.org/r/228124

Change 228124 merged by jenkins-bot:
rl-test: Add instrumentation for User-Agent and Remote IP

https://gerrit.wikimedia.org/r/228124

Change 228173 had a related patch set uploaded (by Krinkle):
rl-test: Fix IP detection to use WebRequest::getIP

https://gerrit.wikimedia.org/r/228173

The planned research has been concluded.

At 26 July 2015 00:09 UTC we changed MediaWiki:Common.js on www.mediawiki.org to make a POST request to rl-test.php?version=10 which logs to /tmp/rl-test.log on mw1017. This file is served through the 'site' module with a short cache expiry (max-age=300).

At 27 July 2015 19:12 UTC (well over 24 hours) this was changed to version=11. The desired outcome is for browsers of recurring visitors to stop using their local cache within 5 minutes, after which they would make an HTTP request with If-None-Match header that no longer returns 304 Not Modified but the updated version.

The majority of browsers stop using the cached version after 5 minutes, but there is a long tail. The v10 drop off took over 24 hours (the change occurs at line 45)

1 count date hour
2 496 26 Jul 2015 00
3 954 26 Jul 2015 01
4 837 26 Jul 2015 02
5 879 26 Jul 2015 03
6 805 26 Jul 2015 04
7 946 26 Jul 2015 05
8 844 26 Jul 2015 06
9 838 26 Jul 2015 07
10 1102 26 Jul 2015 08
11 1074 26 Jul 2015 09
12 1141 26 Jul 2015 10
13 1154 26 Jul 2015 11
14 1309 26 Jul 2015 12
15 1141 26 Jul 2015 13
16 1392 26 Jul 2015 14
17 1317 26 Jul 2015 15
18 1245 26 Jul 2015 16
19 1439 26 Jul 2015 17
20 1255 26 Jul 2015 18
21 1428 26 Jul 2015 19
22 1298 26 Jul 2015 20
23 1237 26 Jul 2015 21
24 1072 26 Jul 2015 22
25 1020 26 Jul 2015 23
26 1112 27 Jul 2015 00
27 1120 27 Jul 2015 01
28 1184 27 Jul 2015 02
29 1224 27 Jul 2015 03
30 1244 27 Jul 2015 04
31 1233 27 Jul 2015 05
32 1775 27 Jul 2015 06
33 1916 27 Jul 2015 07
34 2129 27 Jul 2015 08
35 2487 27 Jul 2015 09
36 2000 27 Jul 2015 10
37 2222 27 Jul 2015 11
38 2590 27 Jul 2015 12
39 2616 27 Jul 2015 13
40 2842 27 Jul 2015 14
41 2751 27 Jul 2015 15
42 2095 27 Jul 2015 16
43 2080 27 Jul 2015 17
44 2251 27 Jul 2015 18
45 658 27 Jul 2015 19
46 3 27 Jul 2015 21
47 1 27 Jul 2015 22
48 2 27 Jul 2015 23
49 4 28 Jul 2015 00
50 11 28 Jul 2015 01
51 5 28 Jul 2015 02
52 3 28 Jul 2015 03
53 5 28 Jul 2015 04
54 5 28 Jul 2015 05
55 22 28 Jul 2015 06
56 5 28 Jul 2015 07
57 5 28 Jul 2015 08
58 7 28 Jul 2015 09
59 1 28 Jul 2015 10
60 9 28 Jul 2015 11
61 1 28 Jul 2015 12
62 2 28 Jul 2015 13
63 1 28 Jul 2015 14
64 6 28 Jul 2015 15
65 1 28 Jul 2015 16
66 8 28 Jul 2015 17
67 1 28 Jul 2015 18
68 2 28 Jul 2015 19
69 6 28 Jul 2015 21
70 1 28 Jul 2015 23
71 1 29 Jul 2015 00
72 1 29 Jul 2015 01
73 6 29 Jul 2015 02
74 1 29 Jul 2015 03
75 1 29 Jul 2015 05
76 6 29 Jul 2015 07
77 2 29 Jul 2015 08
78 1 29 Jul 2015 09
79 4 29 Jul 2015 10
80 1 29 Jul 2015 12
81 1 29 Jul 2015 13
82 1 29 Jul 2015 14
83 2 29 Jul 2015 15
84 1 29 Jul 2015 16
85 1 29 Jul 2015 17
86 3 29 Jul 2015 18
87 1 30 Jul 2015 00
88 2 30 Jul 2015 02
89 1 30 Jul 2015 05
90 1 30 Jul 2015 06
91 1 30 Jul 2015 08
92 2 30 Jul 2015 11
93 2 30 Jul 2015 12
94 1 30 Jul 2015 13
95 2 30 Jul 2015 15
96 1 30 Jul 2015 19

For comparison, the build up of v11:

1 count date hour
2 486 27 Jul 2015 19
3 3104 27 Jul 2015 21
4 2097 27 Jul 2015 22
5 1321 27 Jul 2015 23
6 988 28 Jul 2015 00
7 1183 28 Jul 2015 01
8 1271 28 Jul 2015 02
9 1179 28 Jul 2015 03
10 1047 28 Jul 2015 04
11 1422 28 Jul 2015 05
12 1870 28 Jul 2015 06
13 2221 28 Jul 2015 07
14 2391 28 Jul 2015 08
15 2432 28 Jul 2015 09
16 2119 28 Jul 2015 10
17 2285 28 Jul 2015 11
18 2666 28 Jul 2015 12
19 3071 28 Jul 2015 13
20 3481 28 Jul 2015 14
21 3033 28 Jul 2015 15
22 2558 28 Jul 2015 16
23 1997 28 Jul 2015 17
24 2138 28 Jul 2015 18
25 2053 28 Jul 2015 19
26 2128 28 Jul 2015 20
27 1694 28 Jul 2015 21
28 448 28 Jul 2015 22
29 1058 28 Jul 2015 23
30 1005 29 Jul 2015 00
31 211 29 Jul 2015 01
32 11 29 Jul 2015 02
33 1 29 Jul 2015 03
34 5 29 Jul 2015 04
35 10 29 Jul 2015 05
36 17 29 Jul 2015 06
37 10 29 Jul 2015 07
38 1 29 Jul 2015 08
39 15 29 Jul 2015 09
40 15 29 Jul 2015 10
41 10 29 Jul 2015 11
42 12 29 Jul 2015 12
43 13 29 Jul 2015 13
44 8 29 Jul 2015 14
45 2 29 Jul 2015 15
46 6 29 Jul 2015 16
47 5 29 Jul 2015 17
48 4 29 Jul 2015 18
49 5 29 Jul 2015 19
50 2 29 Jul 2015 20
51 4 29 Jul 2015 21
52 11 29 Jul 2015 22
53 6 30 Jul 2015 00
54 1 30 Jul 2015 01
55 3 30 Jul 2015 02
56 3 30 Jul 2015 03
57 3 30 Jul 2015 07
58 3 30 Jul 2015 08
59 1 30 Jul 2015 09
60 3 30 Jul 2015 10
61 3 30 Jul 2015 11
62 2 30 Jul 2015 12
63 2 30 Jul 2015 15
64 1 30 Jul 2015 16
65 5 30 Jul 2015 18
66 1 30 Jul 2015 19
67 8 30 Jul 2015 20
68 2 30 Jul 2015 22
69 1 30 Jul 2015 23

We ran the experiment a second time after https://gerrit.wikimedia.org/r/227396 was deployed to www.mediawiki.org (adds Cache-Control: must-revalidate header).

At 29 July 2015 01:11 UTC we deployed the version=20 update. And increased to version=21 at 29 July 2015 18:48 UTC (almost 17 hours later). Drop off of v20:

1 count date hour
2 900 29 Jul 2015 01
3 1016 29 Jul 2015 02
4 1080 29 Jul 2015 03
5 974 29 Jul 2015 04
6 1353 29 Jul 2015 05
7 1461 29 Jul 2015 06
8 2297 29 Jul 2015 07
9 2586 29 Jul 2015 08
10 2423 29 Jul 2015 09
11 2007 29 Jul 2015 10
12 2319 29 Jul 2015 11
13 2542 29 Jul 2015 12
14 3030 29 Jul 2015 13
15 3239 29 Jul 2015 14
16 2704 29 Jul 2015 15
17 1993 29 Jul 2015 16
18 2059 29 Jul 2015 17
19 1993 29 Jul 2015 18
20 13 29 Jul 2015 19
21 7 29 Jul 2015 20
22 7 29 Jul 2015 21
23 2 29 Jul 2015 22
24 2 29 Jul 2015 23
25 1 30 Jul 2015 01
26 3 30 Jul 2015 02
27 2 30 Jul 2015 03
28 4 30 Jul 2015 04
29 3 30 Jul 2015 05
30 9 30 Jul 2015 06
31 47 30 Jul 2015 07
32 4 30 Jul 2015 08
33 4 30 Jul 2015 09
34 2 30 Jul 2015 10
35 2 30 Jul 2015 11
36 2 30 Jul 2015 12
37 12 30 Jul 2015 13
38 2 30 Jul 2015 14
39 1 30 Jul 2015 15
40 1 30 Jul 2015 16
41 1 30 Jul 2015 19

Build up of v21:

1 count date hour
2 297 29 Jul 2015 18
3 1907 29 Jul 2015 19
4 1809 29 Jul 2015 20
5 1618 29 Jul 2015 21
6 1127 29 Jul 2015 22
7 1071 29 Jul 2015 23
8 904 30 Jul 2015 00
9 945 30 Jul 2015 01
10 1047 30 Jul 2015 02
11 1063 30 Jul 2015 03
12 887 30 Jul 2015 04
13 1126 30 Jul 2015 05
14 1462 30 Jul 2015 06
15 2061 30 Jul 2015 07
16 2179 30 Jul 2015 08
17 2231 30 Jul 2015 09
18 1958 30 Jul 2015 10
19 2290 30 Jul 2015 11
20 2638 30 Jul 2015 12
21 2713 30 Jul 2015 13
22 3004 30 Jul 2015 14
23 2860 30 Jul 2015 15
24 2137 30 Jul 2015 16
25 1899 30 Jul 2015 17
26 2003 30 Jul 2015 18
27 1823 30 Jul 2015 19
28 1893 30 Jul 2015 20
29 1547 30 Jul 2015 21
30 1317 30 Jul 2015 22
31 950 30 Jul 2015 23
32 215 31 Jul 2015 00

I'm surprised to learn that this did not make a significant difference. There is a similar tail of clients that continue to re-use expired cache. Even now, four days later, there is still a tail of the first experiment (which didn't have must-revalidate headers).

The experiment is continuing today with additional collection of IP and User-Agent information to find any commonalities.

Change 228173 merged by jenkins-bot:
rl-test: Fix IP detection to use WebRequest::getIP

https://gerrit.wikimedia.org/r/228173

Krinkle added a comment.EditedAug 5 2015, 11:44 PM

The third iteration of this experiment confirmed for sure that for a small percentage of users, this supposedly short-lived resource (max-age=300, must-revalidate) is in fact continued to be used for new page views for at least two weeks (we're still seeing a trail of the first and second iterations from last month).

The trail of each iteration is quite constant. This suggests that the affected clients (browsers, proxies or otherwise) have no intention of letting go of their cached copy any time soon.

I've got a copy of the collected IP chains and user-agents but have not been able to find any meaningful commonalities that would help narrow down the cause of the problem.

Here is the complete breakdown of each version per day:

1
2 1 24 Jul 2015 version: -
3 4017 24 Jul 2015 version: 0
4 166 24 Jul 2015 version: 1
5
6 418 25 Jul 2015 version: 0
7 25634 25 Jul 2015 version: 1
8
9 1 26 Jul 2015 version: -
10 4 26 Jul 2015 version: 0
11 25 26 Jul 2015 version: 1
12 26223 26 Jul 2015 version: 10
13
14 5 27 Jul 2015 version: 0
15 10 27 Jul 2015 version: 1
16 37535 27 Jul 2015 version: 10
17 7008 27 Jul 2015 version: 11
18
19 1 28 Jul 2015 version: -
20 4 28 Jul 2015 version: 1
21 111 28 Jul 2015 version: 10
22 46733 28 Jul 2015 version: 11
23
24 1 29 Jul 2015 version: 0
25 5 29 Jul 2015 version: 1
26 33 29 Jul 2015 version: 10
27 1383 29 Jul 2015 version: 11
28 79 29 Jul 2015 version: 12
29 36007 29 Jul 2015 version: 20
30 7829 29 Jul 2015 version: 21
31
32 4 30 Jul 2015 version: -
33 3 30 Jul 2015 version: 1
34 14 30 Jul 2015 version: 10
35 48 30 Jul 2015 version: 11
36 100 30 Jul 2015 version: 20
37 42937 30 Jul 2015 version: 21
38
39 1 31 Jul 2015 version: 0
40 1 31 Jul 2015 version: 1
41 9 31 Jul 2015 version: 10
42 20 31 Jul 2015 version: 11
43 21 31 Jul 2015 version: 20
44 371 31 Jul 2015 version: 21
45 38430 31 Jul 2015 version: 22
46
47 1 01 Aug 2015 version: 10
48 7 01 Aug 2015 version: 11
49 2 01 Aug 2015 version: 20
50 26 01 Aug 2015 version: 21
51 792 01 Aug 2015 version: 22
52 20873 01 Aug 2015 version: 30
53
54 2 02 Aug 2015 version: 1
55 6 02 Aug 2015 version: 10
56 3 02 Aug 2015 version: 20
57 16 02 Aug 2015 version: 21
58 12 02 Aug 2015 version: 22
59 23654 02 Aug 2015 version: 30
60
61 3 03 Aug 2015 version: 1
62 13 03 Aug 2015 version: 10
63 8 03 Aug 2015 version: 11
64 5 03 Aug 2015 version: 20
65 29 03 Aug 2015 version: 21
66 65 03 Aug 2015 version: 22
67 39044 03 Aug 2015 version: 30
68 387 03 Aug 2015 version: 31
69
70 1 04 Aug 2015 version: 0
71 1 04 Aug 2015 version: 1
72 8 04 Aug 2015 version: 10
73 11 04 Aug 2015 version: 11
74 5 04 Aug 2015 version: 20
75 15 04 Aug 2015 version: 21
76 8 04 Aug 2015 version: 22
77 145 04 Aug 2015 version: 30
78 36469 04 Aug 2015 version: 31
79
80 8 05 Aug 2015 version: 11
81 6 05 Aug 2015 version: 20
82 5 05 Aug 2015 version: 21
83 4 05 Aug 2015 version: 22
84 30 05 Aug 2015 version: 30
85 54 05 Aug 2015 version: 31

And here the version promotions:

  • 24-06-2015 05:46 - version=0
  • 24-06-2015 23:47 - version=1
  • 25-06-2015 21:04 - (disabled)
  • 2015-07-26 00:09 - version=10 - Restart without test2.wikipedia.org interference
  • 2015-07-27 19:12 - version=11
  • 2015-07-29 01:10 - version=12 - Confirmation
  • 2015-07-29 01:11 - version=20
  • 2015-07-29 18:48 - version=21
  • 2015-07-31 00:15 - version=22
  • 2015-08-01 00:49 - version=30 - Added logging for request context
  • 2015-08-03 23:40 - version=31
  • 2015-08-04 19:38 - (disabled) - End of experiment

I didn't find any particular common proxies or user-agents in the hits from clients with an expired version at that time.

The majority of expired-hits in fact didn't have any external proxy exposed in the XFF chain (not that this is the raw XFF chain, regardless of which ones we configure as trusted). Suggesting that they are all either directly connecting to us or through a proxy that doesn't provide XFF.

The User-Agent are not notably different from regular traffic patterns. Here are the most common user-agent strings from the last 24 hours of hits (all from at least an hour after the experiment ended).

The request context (skin, language, username-or-anon) also didn't yield anything useful. It affects both logged-in and logged-out users.

1 Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.17
2 Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
3 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
4 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:39.0) Gecko/20100101 Firefox/39.0
5 Mozilla/5.0 (X11; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0
6 Mozilla/5.0 (X11; Linux x86_64; rv:39.0) Gecko/20100101 Firefox/39.0
7 Mozilla/5.0 (X11; Linux x86_64; rv:37.0) Gecko/20100101 Firefox/37.0 Iceweasel/37.0.1
8 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/43.0.2357.130 Chrome/43.0.2357.130 Safari/537.36
9 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/37.0.2062.120 Chrome/37.0.2062.120 Safari/537.36
10 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.* Safari/537.36
11 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.* Safari/537.36
12 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.* Safari/537.36
13 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36
14 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
15 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.34 Safari/534.24
16 Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:39.0) Gecko/20100101 Firefox/39.0
17 Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
18 Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; MAARJS; rv:11.0) like Gecko
19 Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
20 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
21 Mozilla/5.0 (Windows NT 6.3; Trident/7.0; Touch; MALCJS; rv:11.0) like Gecko
22 Mozilla/5.0 (Windows NT 6.1; rv:39.0) Gecko/20100101 Firefox/39.0
23 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
24 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0
25 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.10) Gecko/20100101 Firefox/10.0.10
26 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
27 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
28 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.107 Safari/537.36
29 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36
30 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
31 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
32 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
33 Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
34 Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Firefox/39.0
35 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
36 Mozilla/5.0 (Windows NT 10.0; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
37 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
38 Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240
39 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36
40 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36
41 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36
42 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36
43 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Firefox/38.0
44 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0
45 Mozilla/5.0 (Linux; U; Android 4.2.2; pt-pt; HUAWEI Y320-U10 Build/HUAWEIY320-U10) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
46 Mozilla/5.0 (Linux; Android 4.4.2; EverGlow D35 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36
47 Mozilla/5.0 (Linux; Android 4.1.1; HTL21 Build/JRO03C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.81 Mobile Safari/537.36 OPR/28.0.1764.90386

Conclusion: We're keeping the must-revalidate header. Although it remains a mystery why some clients are seemingly holding on and making use of an expired resource for far too long.

Krinkle removed Krinkle as the assignee of this task.Aug 6 2015, 6:08 AM
Krinkle moved this task from Assigned to Backlog on the MediaWiki-ResourceLoader board.
Krinkle moved this task from Doing to Backlog: Small & Maintenance on the Performance-Team board.
Gilles added a comment.Aug 6 2015, 7:37 AM

Frozen/malfunctioning/completely off computer clocks, perhaps? I remember an era where cracks for running pirated software for high-end stuff (autocad and the like) would essentially freeze the clock to circumvent copy protection software. I also know from other experiments on clients' local time that a surprisingly high portion of people have their computer's date set completely wrong.

Krinkle closed this task as Resolved.Aug 21 2015, 2:56 PM
Krinkle claimed this task.
Krinkle moved this task from Backlog to Assigned on the MediaWiki-ResourceLoader board.