Page MenuHomePhabricator

The 5min expiry for load.php/startup should be relative to request time instead of cache time
Closed, ResolvedPublic

Assigned To
Authored By
Catrope
Jul 12 2015, 11:12 PM
Referenced Files
F28474111: export-rate.png
Mar 26 2019, 7:41 PM
F28474151: per100s.png
Mar 26 2019, 7:41 PM
F28474114: export-status.png
Mar 26 2019, 7:41 PM
F28474152: per10s.png
Mar 26 2019, 7:41 PM
F28400274: 10s-rate-eqiad.png
Mar 16 2019, 8:04 PM
F28400247: eqiad-midday.png
Mar 16 2019, 8:04 PM
F28400213: traffic-drain.png
Mar 16 2019, 8:04 PM
F28400299: 100s-rate.png
Mar 16 2019, 8:04 PM
Tokens
"Yellow Medal" token, awarded by Gilles.

Description

$ curl -I 'https://en.wikipedia.org/w/load.php?modules=startup&only=scripts&skin=vector&*'
HTTP/1.1 200 OK
Server: nginx/1.9.2
Date: Sun, 12 Jul 2015 22:44:22 GMT
Content-Type: text/javascript; charset=utf-8
Connection: keep-alive
X-Powered-By: HHVM/3.6.1
X-Content-Type-Options: nosniff
Cache-control: public, max-age=300, s-maxage=300
Vary: Accept-Encoding
Expires: Sun, 12 Jul 2015 22:49:22 GMT
ETag: W/"b7xnhKt7"
X-Varnish: 371412287, 1198588436
Via: 1.1 varnish, 1.1 varnish
Age: 0
X-Cache: cp1054 miss (0), cp1067 frontend miss (0)
Strict-Transport-Security: max-age=31536000
Set-Cookie: GeoIP=:::::v6; Path=/; Domain=.wikipedia.org
X-Analytics: https=1
Set-Cookie: WMF-Last-Access=12-Jul-2015;Path=/;HttpOnly;Expires=Thu, 13 Aug 2015 12:00:00 GMT

(wait a few minutes)

$ curl -I 'https://en.wikipedia.org/w/load.php?modules=startup&only=scripts&skin=vector&*'
HTTP/1.1 200 OK
Server: nginx/1.9.2
Date: Sun, 12 Jul 2015 22:47:46 GMT
Content-Type: text/javascript; charset=utf-8
Connection: keep-alive
X-Powered-By: HHVM/3.6.1
X-Content-Type-Options: nosniff
Cache-control: public, max-age=300, s-maxage=300
Vary: Accept-Encoding
Expires: Sun, 12 Jul 2015 22:49:22 GMT
ETag: W/"b7xnhKt7"
X-Varnish: 371412287, 1199144708 1198588436
Via: 1.1 varnish, 1.1 varnish
Age: 204
X-Cache: cp1054 miss (0), cp1067 frontend hit (7)
Strict-Transport-Security: max-age=31536000
Set-Cookie: GeoIP=:::::v6; Path=/; Domain=.wikipedia.org
X-Analytics: https=1
Set-Cookie: WMF-Last-Access=12-Jul-2015;Path=/;HttpOnly;Expires=Thu, 13 Aug 2015 12:00:00 GMT

In the first response, the Expires timestamp is exactly 5 minutes (=300 seconds, and Cache-Control has max-age=300) after the Date timestamp. It's also clear from the X-Cache header that the first response is a cache miss (even though I used the most popular load.php URL in existence; but it seems few people hit it without an Accept-Encoding header).

In the second response, the Expires timestamp is the same as in the first response, and is much less than 5 minutes after the Date timestamp. According to the X-Cache header it's a cache hit (as you can also tell from the X-Cache header, I sent 6 more requests in between that are not shown here). So it appears that the Expires timestamp is part of the data that is cached, and will frequently be stricter than the Cache-Control header.

This would be a problem if clients obey the Expires header instead of the Cache-Control header in cases like these where the Expires header is stricter. Clients would cache responses for shorter periods of time than we intend, and their revalidations (INM/304 cycles) would not be randomly distributed over time, but would be clustered right after the timestamp in the Expires header. It seems at least Chrome does in fact behave this way: I have seen it send an INM request for a cached response whose Expires header had expired even though the max-age duration had not yet expired.

I suspect this happens because we explicitly set the Expires header in ResourceLoader. We should consider not doing this, and letting Varnish compute it instead (if we decide we need Expires at all; how many clients don't understand Cache-Control these days?).

Event Timeline

Catrope raised the priority of this task from to Needs Triage.
Catrope updated the task description. (Show Details)
Catrope added a project: Performance-Team.
Catrope subscribed.
Krinkle moved this task from Inbox to Accepted Enhancement on the MediaWiki-ResourceLoader board.
Krinkle renamed this task from Expires header for load.php responses is relative to caching timestamp, not request timestamp to Expires header for load.php should be relative to request time intead of cache time.Sep 4 2015, 3:56 AM
Krinkle renamed this task from Expires header for load.php should be relative to request time intead of cache time to Expires header for load.php should be relative to request time instead of cache time.
Krinkle added a project: Performance-Team.

If you are using a CDN I recommend to use Cache-Control [..]. This prevents request-peaks to your origin-server. With "Expires" all browsers will request you at the same time.

This supports the suspicion that it causes peaks.

Expires

Back in the day, this was the standard way [..] . It is still fairly useful for older user agents, [..]. On most modern systems, the "cache-control" headers "max-age" and "s-maxage" will take precedence, [..]

According to this article, though, max-age does take precedence.

@Catrope wrote:

It seems at least Chrome does in fact behave this way: I have seen it send an INM request for a cached response whose Expires header had expired even though the max-age duration had not yet expired.

Micro tasks:

  • Confirm that our edge (Varnish and Ngninx) does indeed support adding Expires automatically. And confirm that Apache (backend) will not do so.
  • Confirm whether the above behaviour is still the case on latest Firefox and Chrome. If so, involve upstream on what the expected behaviour is.
  • If edge supports it and the behaviour isn't changing soon, let's remove it from MediaWiki when UseSquid is enabled.

Expires

Back in the day, this was the standard way [..] . It is still fairly useful for older user agents, [..]. On most modern systems, the "cache-control" headers "max-age" and "s-maxage" will take precedence, [..]

According to this article, though, max-age does take precedence.

The article is from 2013, and my bug report is from 2015. Still, it's worth confirming that this still happens in the latest versions of FF/Chrome.

  • Confirm that our edge (Varnish and Ngninx) does indeed support adding Expires automatically. And confirm that Apache (backend) will not do so.

Worst come to worst, we could have the edge overwrite Expires headers in certain cases (e.g. a CC header is present and it's a load.php request).

  • Confirm whether the above behaviour is still the case on latest Firefox and Chrome. If so, involve upstream on what the expected behaviour is.

One possibility that's unhelpful for us but logical from upstream's perspective: the most restrictive header is followed, so if Expires and Cache-Control disagree, then whichever expiry comes first is observed. expiry = min(Expires, now+maxage)

  • If edge supports it and the behaviour isn't changing soon, let's remove it from MediaWiki when UseSquid is enabled.

I don't think that's a good idea, because it'd cause 3rd parties with UseSquid enabled to lose Expires headers unless they also set up the same magic as WMF. I don't know how unsafe it is to send responses with only a CC header and no Expires header, but if we're trying to make Expires be correct (as opposed to just dropping it), presumably there's a reason for that.

  • If edge supports it and the behaviour isn't changing soon, let's remove it from MediaWiki when UseSquid is enabled.

I don't think that's a good idea, because it'd cause 3rd parties with UseSquid enabled to lose Expires headers unless they also set up the same magic as WMF.

By "If edge supports it" I meant if any caching proxy we're willing to support can easily add it in a non-buggy way. Either by default, with a built-in config option, or widely available plugin.

I'm not proposing there be any end-user request without Expires header. For stock MediaWiki, third-parties with UseSquid, or Wikimedia.

This bug is very specific to use of a caching proxy. Apart from removing Expires entirely, I don't know of a consistent solution other than flagging with UseSquid and the proxy adding it back.

The article is from 2013, and my bug report is from 2015. Still, it's worth confirming that this still happens in the latest versions of FF/Chrome.

Confirmed this is still happening in Chrome 52.

I haven't really looked at this ticket before, so bear with me starting back at the top...

In the original second example, there is no conflict or mismatch between the information in Cache-Control and Expires, because there's also an Age header which indicates the existing age of the object. The relevant headers in the second request are:

Date: Sun, 12 Jul 2015 22:47:46 GMT
Cache-control: public, max-age=300, s-maxage=300
Expires: Sun, 12 Jul 2015 22:49:22 GMT
Age: 204

The Age header indicates 204 of the 300 seconds have already passed, leaving 96 seconds left on the validity of this object, which is the same as the number of seconds between the timestamps in Date and Expires.

The fact that such a hot object expires every 5 minutes is bad in general - we should find ways to lengthen that time (I'm pretty sure we're already working on that elsewhere with versioning?). Aside from having a ridiculously short TTL for such a popular object, there's nothing fundamentally wrong with how the caching behaves for this object.

I synced with @BBlack on IRC just now and resolved the confusion @Catrope and I were having.

13.2.4 Expiration Calculations
The max-age directive takes priority over Expires, so if max-age is present in a response, the calculation is simply:

freshness_lifetime = max_age_value

Otherwise, if Expires is present in the response, the calculation is:

freshness_lifetime = expires_value - date_value

Given this, Roan and I thought Chrome and Firefox were misbehaving due to the object clearly expiring at the time of the Expires header - instead of 5 minutes after the browser received the object.

However, we forgot about the Age header.

13.2.4 Expiration Calculations (continued..)

response_is_fresh = (freshness_lifetime > current_age)

13.2.3 Age Calculations

corrected_received_age = max(now - date_value, age_value)

[more edge cases and adjustments..]

The browser probably wasn't considering the Expires header. The "correct" expiry using max-age doesn't start at 0 for the browser, it starts at Age. And so, naturally, the correct expiry happens to match our explicit Expires header - after all, we create them in our code using the same offset.

Cache-Control max-age essentially "counts down" by first deducting the age accumulated thus far. This is quite reasonable considering Cache-Control "max-age" is basically a port of the previous absolute time behaviour; but using relative numbers. The underlying intent to expire an object at a predictable moment in time hasn't changed, and that's probably a good thing.

Back to the original issue: This means our assumption was wrong. This is not a bug caused by the Expires header. It will happen the same way without an Expires header. If we want to fix this, we'd have to manipulate the Age header as well. Which is most probably something that requires custom configuration that isn't commonly available to third-party site admins.

On the other hand, if you take Varnish (and other caching proxies) out of the picture and resort to plain MediaWiki with file caching, this bug doesn't happen if you remove the Expires header .. because we don't set the Age header in MediaWiki.

@Catrope: Brandon also mentioned that in practice the stampede isn't too bad (even at our scale) because browser's only request it after 5 minutes when the next page view occurs, which is fairly distributed already.

If we still want to tackle this, we can consider hacking with the Age header in Varnish for Wikimedia specifically and document it as a scaling/performance tuning hack for interested third-parties.

Alternatively, we could evaluate other Cache-Control mechanisms and Varnish configuration options relating to the grace window during which stale cache may be used.

Or maybe we can bump the startup module expiry to 10 minutes across the board. Personally I'd like to avoid raising the expiry at the moment as it hurts deployments, as well as updates propagating to users when they make changes to gadgets - in ways that is quite untransparent to reason about.

We could investigate purging to some extend, but that gets out of hand very quickly. Right now that would require scap to send some 1.3 million purges on every deployment (800 wikis * 419 language codes * 4 skins). The current rate of purges from MediaWiki is already problematic (Brandon suggests we'd have to spread it at 1000 purges per second).

Or maybe we can bump the startup module expiry to 10 minutes across the board.

Actually, no. The main use case I want to address here is that clients should not need to go back to the server for the startup module within 5 minutes of first retrieving it - regardless of how old the response from Varnish was. Raising the max age to 10min would just move the problem. Now users who start a page view session in the 9th minute will be forced to go for a 304 response after only 1 minute.

Instead, it seems like the only solution here is to omit the Age header in Varnish for load.php (or override it to Age: 0). @Gilles wants to experiment with this in Vagrant. Seems like it might be a simple VCL patch.

In an attempt to verify whether or not we can observe there being more startup requests around the 5min mark, I ran a few log tails on a Varnish.

# Gather 100,000 requests for the default startup module url from 1 varnish in the text cluster
krinkle@cp1066:~$ varnishlog -q 'ReqURL ~ "load.php.debug=false&lang=en&modules=startup&only=scripts&skin=vector" and ReqMethod eq "GET"' -n frontend -i RespHeader -i ReqURL -i ReqMethod -i Timestamp -C -k 100000 | grep -E 'Timestamp\s+Req' > startup-en.log

krinkle@cp1066:~$ wc -l startup-en.log 
100000 startup-en.log 

krinkle@cp1066:~$ head startup-en.log 
-   Timestamp      Req: 1481859976.317621 0.000000 0.000000
-   Timestamp      Req: 1481859976.320093 0.000000 0.000000
-   Timestamp      Req: 1481859976.335609 0.000000 0.000000

# count and group by 100 seconds
krinkle@cp1066:~$ cat startup-en.log | cut -d'.' -f1 | cut -c '25-32' | xargs -d '\n' -P 10 -I % date --date='@%00' | sort | uniq -c
    955 Fri Dec 16 03:45:00 UTC 2016
   3333 Fri Dec 16 03:46:40 UTC 2016
   3142 Fri Dec 16 03:48:20 UTC 2016
   4095 Fri Dec 16 03:50:00 UTC 2016
   3379 Fri Dec 16 03:51:40 UTC 2016
   3081 Fri Dec 16 03:53:20 UTC 2016
   3935 Fri Dec 16 03:55:00 UTC 2016
   3414 Fri Dec 16 03:56:40 UTC 2016
   3025 Fri Dec 16 03:58:20 UTC 2016
   3971 Fri Dec 16 04:00:00 UTC 2016
   3400 Fri Dec 16 04:01:40 UTC 2016
   2961 Fri Dec 16 04:03:20 UTC 2016
   3850 Fri Dec 16 04:05:00 UTC 2016
   3254 Fri Dec 16 04:06:40 UTC 2016
   2846 Fri Dec 16 04:08:20 UTC 2016
   3744 Fri Dec 16 04:10:00 UTC 2016
   3062 Fri Dec 16 04:11:40 UTC 2016
   2757 Fri Dec 16 04:13:20 UTC 2016
   3727 Fri Dec 16 04:15:00 UTC 2016
   3157 Fri Dec 16 04:16:40 UTC 2016
   2708 Fri Dec 16 04:18:20 UTC 2016
   3613 Fri Dec 16 04:20:00 UTC 2016
   3074 Fri Dec 16 04:21:40 UTC 2016
   2758 Fri Dec 16 04:23:20 UTC 2016
   3625 Fri Dec 16 04:25:00 UTC 2016
   2972 Fri Dec 16 04:26:40 UTC 2016
   2578 Fri Dec 16 04:28:20 UTC 2016
   3388 Fri Dec 16 04:30:00 UTC 2016
   2869 Fri Dec 16 04:31:40 UTC 2016
   2637 Fri Dec 16 04:33:20 UTC 2016
   3479 Fri Dec 16 04:35:00 UTC 2016
   1211 Fri Dec 16 04:36:40 UTC 2016

plit2.png (704×1 px, 38 KB)

Looks like indeed every 5th minute has consistently about 30% more requests in it (4000 vs 3100, later 3400 vs 2600). I didn't expect it to be observable necessarily since this request is driven by a user actively making a page view, so it's naturally quite distributed. Unless a user requests a page exactly at the point the cache expired, their browser will not make such request around that time. And as mentioned, the problem here is not that we get stampedes (which we handle just fine). The problem is the negative performance impact on users caused by them having to re-request the startup module on subsequent page views within the first 5 minutes of their session.

Keeping the max-age at 5 mins and forcing Age: 0 sounds good to me. To respond to @BBlack's comment about this hot object having such a short expiry: RL's caching strategy works by having one object (they"module manifest" or "startup module") have a short expiry so that (almost) all other objects can have long expiries but still not be out of date for long. It does this by having the manifest list the freshness status of every other module. This manifest is hot (because it's always need to load anything else) and has a 5-minute expiry (so that changes propagate quickly). When a change occurs, the hash in the manifest entry for that module changes, clients get the new manifest within 5 minutes, and they use this hash to change cache-busting query string parameter when requesting the changed module. The response to that request will have a 30-day expiry.

Change 330874 had a related patch set uploaded (by Gilles):
Set Age: 0 for RL requests served by Varnish

https://gerrit.wikimedia.org/r/330874

Change 330874 merged by jenkins-bot:
Set Age: 0 for RL requests served by Varnish

https://gerrit.wikimedia.org/r/330874

@BBlack can you take a look at the Vagrant VCL patch above (very simple) and see if that's a change you would be comfortable with in production?

@BBlack I'd like to apply a similar patch to prod puppet. First cherry-picked on beta for sanity, however this won't help confirm our hypothesis due to lack of natural traffic. After confirming in beta, I'd like to see it applied to a production varnish-frontend for an hour or so during which I can run the same varnishlog as before. And once confirmed that the distribution is how we want, we can roll it out everywhere.

I can't see any negative impact that may be possible. A browser will either make a request for the needed load.php url or it will use its http cache (if it has a satisfying entry). Worse case scenario it might be more biased towards the former if somehow there is an yet-to-be-discovererd bug in some browsers if it sees Age: 0 more often than currently the case, in which case neither their nor our performance would be significantly impacted but we'd simply decide to roll back.

Krinkle changed the task status from Open to Stalled.Feb 7 2018, 1:29 AM

@Krinkle - Seems sane to try. I do wonder if browsers will actually allow Age: 0 to imply a longer lifetime than the Expires header does. It seems like they should, but some may simply take the most pessimistic of the two indicators.

Krinkle changed the task status from Stalled to Open.Feb 7 2018, 2:23 AM

@BBlack Thanks for the quick response. I'll put move it out of blocked then. Will let you know when I'm ready to try this (might be a few weeks, given other priorities).

Change 496497 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: set /w/load.php Age to 0

https://gerrit.wikimedia.org/r/496497

I've re-run the analysis from 2 years ago (T105657#2918703) to see what behaviour is today, whether it's still the same, and to have a better baseline to see the impact of the above patch when it rolls out next week.

NOTE: I tried to do this analysis late Friday but found that the daily seasonality at that time was making it impossible to detect any useful patterns in the traffic. This is because (like every day) in the evening UTC, the traffic "drains" from Europe at such an incredible rate that any pattern within a 15 minute range is completely obliterated by the fact that 80% of the traffic disappears over the course of an hour or so… 😅
traffic-drain.png (1×954 px, 42 KB)
From grafana.wikimedia.org
Edge traffic for /w/load.php to the Esams cluster in Amsterdam.

I've re-run this around mid-day instead where there is a more stable traffic rate.

eqiad-midday.png (1×1 px, 108 KB)
From grafana.wikimedia.org
Edge traffic for /w/load.php to the Eqiad cluster (USA).

So, here it is.

# Gather 200,000 requests from 1 varnish instance in Eqiad (~2h).
krinkle@cp1085:~$ varnishlog -q 'ReqURL ~ "load.php\?.*lang=en&modules=startup&only=scripts&skin=vector.*" and ReqMethod eq "GET"' -n frontend -i RespHeader -i ReqURL -i ReqMethod -i Timestamp -C -k 200000 | grep -E 'Timestamp\s+Req' > startup-en.log
# Review
$ wc -l startup-en.log; head startup-en.log
300000 startup-en.log
-   Timestamp      Req: 1552699693.##### 0.000000 0.000000
# Group
$ cat startup-en.log | cut -d'.' -f1 | cut -c '25-33' | xargs -d '\n' -P 30 -I % date --date='@%0' | sort | uniq -c
Grouped by 10s
231 Sat Mar 16 16:37:20 UTC 2019
253 Sat Mar 16 16:37:30 UTC 2019
313 Sat Mar 16 16:37:40 UTC 2019
252 Sat Mar 16 16:37:50 UTC 2019
230 Sat Mar 16 16:38:00 UTC 2019
279 Sat Mar 16 16:38:10 UTC 2019
243 Sat Mar 16 16:38:20 UTC 2019
249 Sat Mar 16 16:38:30 UTC 2019
267 Sat Mar 16 16:38:40 UTC 2019
287 Sat Mar 16 16:38:50 UTC 2019
258 Sat Mar 16 16:39:00 UTC 2019
256 Sat Mar 16 16:39:10 UTC 2019
233 Sat Mar 16 16:39:20 UTC 2019
279 Sat Mar 16 16:39:30 UTC 2019
256 Sat Mar 16 16:39:40 UTC 2019
239 Sat Mar 16 16:39:50 UTC 2019
251 Sat Mar 16 16:40:00 UTC 2019
250 Sat Mar 16 16:40:10 UTC 2019
250 Sat Mar 16 16:40:20 UTC 2019
260 Sat Mar 16 16:40:30 UTC 2019
230 Sat Mar 16 16:40:40 UTC 2019
242 Sat Mar 16 16:40:50 UTC 2019
269 Sat Mar 16 16:41:00 UTC 2019
213 Sat Mar 16 16:41:10 UTC 2019
244 Sat Mar 16 16:41:20 UTC 2019
241 Sat Mar 16 16:41:30 UTC 2019
228 Sat Mar 16 16:41:40 UTC 2019
243 Sat Mar 16 16:41:50 UTC 2019
221 Sat Mar 16 16:42:00 UTC 2019
211 Sat Mar 16 16:42:10 UTC 2019
253 Sat Mar 16 16:42:20 UTC 2019
245 Sat Mar 16 16:42:30 UTC 2019
321 Sat Mar 16 16:42:40 UTC 2019
274 Sat Mar 16 16:42:50 UTC 2019
314 Sat Mar 16 16:43:00 UTC 2019
321 Sat Mar 16 16:43:10 UTC 2019
266 Sat Mar 16 16:43:20 UTC 2019
257 Sat Mar 16 16:43:30 UTC 2019
268 Sat Mar 16 16:43:40 UTC 2019
232 Sat Mar 16 16:43:50 UTC 2019
256 Sat Mar 16 16:44:00 UTC 2019
240 Sat Mar 16 16:44:10 UTC 2019
253 Sat Mar 16 16:44:20 UTC 2019
256 Sat Mar 16 16:44:30 UTC 2019
239 Sat Mar 16 16:44:40 UTC 2019
236 Sat Mar 16 16:44:50 UTC 2019
237 Sat Mar 16 16:45:00 UTC 2019
191 Sat Mar 16 16:45:10 UTC 2019
245 Sat Mar 16 16:45:20 UTC 2019
251 Sat Mar 16 16:45:30 UTC 2019
226 Sat Mar 16 16:45:40 UTC 2019
217 Sat Mar 16 16:45:50 UTC 2019
241 Sat Mar 16 16:46:00 UTC 2019
224 Sat Mar 16 16:46:10 UTC 2019
221 Sat Mar 16 16:46:20 UTC 2019
212 Sat Mar 16 16:46:30 UTC 2019
203 Sat Mar 16 16:46:40 UTC 2019
227 Sat Mar 16 16:46:50 UTC 2019
246 Sat Mar 16 16:47:00 UTC 2019
256 Sat Mar 16 16:47:10 UTC 2019
255 Sat Mar 16 16:47:20 UTC 2019
304 Sat Mar 16 16:47:30 UTC 2019
306 Sat Mar 16 16:47:40 UTC 2019
236 Sat Mar 16 16:47:50 UTC 2019
309 Sat Mar 16 16:48:00 UTC 2019
280 Sat Mar 16 16:48:10 UTC 2019
272 Sat Mar 16 16:48:20 UTC 2019
252 Sat Mar 16 16:48:30 UTC 2019
285 Sat Mar 16 16:48:40 UTC 2019
257 Sat Mar 16 16:48:50 UTC 2019
242 Sat Mar 16 16:49:00 UTC 2019
272 Sat Mar 16 16:49:10 UTC 2019
238 Sat Mar 16 16:49:20 UTC 2019
228 Sat Mar 16 16:49:30 UTC 2019
236 Sat Mar 16 16:49:40 UTC 2019
232 Sat Mar 16 16:49:50 UTC 2019
219 Sat Mar 16 16:50:00 UTC 2019
224 Sat Mar 16 16:50:10 UTC 2019
239 Sat Mar 16 16:50:20 UTC 2019
251 Sat Mar 16 16:50:30 UTC 2019
239 Sat Mar 16 16:50:40 UTC 2019
223 Sat Mar 16 16:50:50 UTC 2019
204 Sat Mar 16 16:51:00 UTC 2019
240 Sat Mar 16 16:51:10 UTC 2019
224 Sat Mar 16 16:51:20 UTC 2019
239 Sat Mar 16 16:51:30 UTC 2019
221 Sat Mar 16 16:51:40 UTC 2019
235 Sat Mar 16 16:51:50 UTC 2019
237 Sat Mar 16 16:52:00 UTC 2019
225 Sat Mar 16 16:52:10 UTC 2019
247 Sat Mar 16 16:52:20 UTC 2019
310 Sat Mar 16 16:52:30 UTC 2019
304 Sat Mar 16 16:52:40 UTC 2019
276 Sat Mar 16 16:52:50 UTC 2019
338 Sat Mar 16 16:53:00 UTC 2019
242 Sat Mar 16 16:53:10 UTC 2019
264 Sat Mar 16 16:53:20 UTC 2019
259 Sat Mar 16 16:53:30 UTC 2019
256 Sat Mar 16 16:53:40 UTC 2019
269 Sat Mar 16 16:53:50 UTC 2019
233 Sat Mar 16 16:54:00 UTC 2019
252 Sat Mar 16 16:54:10 UTC 2019
228 Sat Mar 16 16:54:20 UTC 2019
252 Sat Mar 16 16:54:30 UTC 2019
224 Sat Mar 16 16:54:40 UTC 2019
250 Sat Mar 16 16:54:50 UTC 2019
249 Sat Mar 16 16:55:00 UTC 2019
232 Sat Mar 16 16:55:10 UTC 2019
243 Sat Mar 16 16:55:20 UTC 2019
249 Sat Mar 16 16:55:30 UTC 2019
209 Sat Mar 16 16:55:40 UTC 2019
218 Sat Mar 16 16:55:50 UTC 2019
218 Sat Mar 16 16:56:00 UTC 2019
201 Sat Mar 16 16:56:10 UTC 2019
224 Sat Mar 16 16:56:20 UTC 2019
205 Sat Mar 16 16:56:30 UTC 2019
235 Sat Mar 16 16:56:40 UTC 2019
220 Sat Mar 16 16:56:50 UTC 2019
224 Sat Mar 16 16:57:00 UTC 2019
195 Sat Mar 16 16:57:10 UTC 2019
288 Sat Mar 16 16:57:20 UTC 2019
284 Sat Mar 16 16:57:30 UTC 2019
287 Sat Mar 16 16:57:40 UTC 2019
301 Sat Mar 16 16:57:50 UTC 2019
291 Sat Mar 16 16:58:00 UTC 2019
277 Sat Mar 16 16:58:10 UTC 2019
274 Sat Mar 16 16:58:20 UTC 2019
255 Sat Mar 16 16:58:30 UTC 2019
250 Sat Mar 16 16:58:40 UTC 2019
250 Sat Mar 16 16:58:50 UTC 2019
253 Sat Mar 16 16:59:00 UTC 2019
232 Sat Mar 16 16:59:10 UTC 2019
256 Sat Mar 16 16:59:20 UTC 2019
220 Sat Mar 16 16:59:30 UTC 2019
239 Sat Mar 16 16:59:40 UTC 2019
290 Sat Mar 16 16:59:50 UTC 2019
237 Sat Mar 16 17:00:00 UTC 2019
250 Sat Mar 16 17:00:10 UTC 2019
263 Sat Mar 16 17:00:20 UTC 2019
238 Sat Mar 16 17:00:30 UTC 2019
228 Sat Mar 16 17:00:40 UTC 2019
216 Sat Mar 16 17:00:50 UTC 2019
235 Sat Mar 16 17:01:00 UTC 2019
239 Sat Mar 16 17:01:10 UTC 2019
223 Sat Mar 16 17:01:20 UTC 2019
216 Sat Mar 16 17:01:30 UTC 2019
219 Sat Mar 16 17:01:40 UTC 2019
243 Sat Mar 16 17:01:50 UTC 2019
245 Sat Mar 16 17:02:00 UTC 2019
203 Sat Mar 16 17:02:10 UTC 2019
301 Sat Mar 16 17:02:20 UTC 2019
267 Sat Mar 16 17:02:30 UTC 2019
295 Sat Mar 16 17:02:40 UTC 2019
296 Sat Mar 16 17:02:50 UTC 2019
289 Sat Mar 16 17:03:00 UTC 2019
283 Sat Mar 16 17:03:10 UTC 2019
283 Sat Mar 16 17:03:20 UTC 2019
277 Sat Mar 16 17:03:30 UTC 2019
261 Sat Mar 16 17:03:40 UTC 2019
269 Sat Mar 16 17:03:50 UTC 2019
245 Sat Mar 16 17:04:00 UTC 2019
242 Sat Mar 16 17:04:10 UTC 2019
257 Sat Mar 16 17:04:20 UTC 2019
276 Sat Mar 16 17:04:30 UTC 2019
250 Sat Mar 16 17:04:40 UTC 2019
244 Sat Mar 16 17:04:50 UTC 2019
254 Sat Mar 16 17:05:00 UTC 2019
230 Sat Mar 16 17:05:10 UTC 2019
276 Sat Mar 16 17:05:20 UTC 2019
237 Sat Mar 16 17:05:30 UTC 2019
234 Sat Mar 16 17:05:40 UTC 2019
240 Sat Mar 16 17:05:50 UTC 2019
234 Sat Mar 16 17:06:00 UTC 2019
222 Sat Mar 16 17:06:10 UTC 2019
225 Sat Mar 16 17:06:20 UTC 2019
224 Sat Mar 16 17:06:30 UTC 2019
250 Sat Mar 16 17:06:40 UTC 2019
200 Sat Mar 16 17:06:50 UTC 2019
235 Sat Mar 16 17:07:00 UTC 2019
314 Sat Mar 16 17:07:10 UTC 2019
300 Sat Mar 16 17:07:20 UTC 2019
283 Sat Mar 16 17:07:30 UTC 2019
286 Sat Mar 16 17:07:40 UTC 2019
283 Sat Mar 16 17:07:50 UTC 2019
302 Sat Mar 16 17:08:00 UTC 2019
294 Sat Mar 16 17:08:10 UTC 2019
263 Sat Mar 16 17:08:20 UTC 2019
289 Sat Mar 16 17:08:30 UTC 2019
271 Sat Mar 16 17:08:40 UTC 2019
272 Sat Mar 16 17:08:50 UTC 2019
277 Sat Mar 16 17:09:00 UTC 2019
254 Sat Mar 16 17:09:10 UTC 2019
249 Sat Mar 16 17:09:20 UTC 2019
248 Sat Mar 16 17:09:30 UTC 2019
257 Sat Mar 16 17:09:40 UTC 2019
249 Sat Mar 16 17:09:50 UTC 2019
248 Sat Mar 16 17:10:00 UTC 2019
247 Sat Mar 16 17:10:10 UTC 2019
220 Sat Mar 16 17:10:20 UTC 2019
260 Sat Mar 16 17:10:30 UTC 2019
249 Sat Mar 16 17:10:40 UTC 2019
239 Sat Mar 16 17:10:50 UTC 2019
245 Sat Mar 16 17:11:00 UTC 2019
248 Sat Mar 16 17:11:10 UTC 2019
253 Sat Mar 16 17:11:20 UTC 2019
225 Sat Mar 16 17:11:30 UTC 2019
235 Sat Mar 16 17:11:40 UTC 2019
252 Sat Mar 16 17:11:50 UTC 2019
281 Sat Mar 16 17:12:00 UTC 2019
283 Sat Mar 16 17:12:10 UTC 2019
279 Sat Mar 16 17:12:20 UTC 2019
264 Sat Mar 16 17:12:30 UTC 2019
272 Sat Mar 16 17:12:40 UTC 2019
287 Sat Mar 16 17:12:50 UTC 2019
301 Sat Mar 16 17:13:00 UTC 2019
264 Sat Mar 16 17:13:10 UTC 2019
256 Sat Mar 16 17:13:20 UTC 2019
275 Sat Mar 16 17:13:30 UTC 2019
293 Sat Mar 16 17:13:40 UTC 2019
261 Sat Mar 16 17:13:50 UTC 2019
269 Sat Mar 16 17:14:00 UTC 2019
264 Sat Mar 16 17:14:10 UTC 2019
250 Sat Mar 16 17:14:20 UTC 2019
257 Sat Mar 16 17:14:30 UTC 2019
269 Sat Mar 16 17:14:40 UTC 2019
255 Sat Mar 16 17:14:50 UTC 2019
254 Sat Mar 16 17:15:00 UTC 2019
247 Sat Mar 16 17:15:10 UTC 2019
263 Sat Mar 16 17:15:20 UTC 2019
236 Sat Mar 16 17:15:30 UTC 2019
266 Sat Mar 16 17:15:40 UTC 2019
238 Sat Mar 16 17:15:50 UTC 2019
213 Sat Mar 16 17:16:00 UTC 2019
263 Sat Mar 16 17:16:10 UTC 2019
235 Sat Mar 16 17:16:20 UTC 2019
225 Sat Mar 16 17:16:30 UTC 2019
194 Sat Mar 16 17:16:40 UTC 2019
236 Sat Mar 16 17:16:50 UTC 2019
280 Sat Mar 16 17:17:00 UTC 2019
313 Sat Mar 16 17:17:10 UTC 2019
265 Sat Mar 16 17:17:20 UTC 2019
280 Sat Mar 16 17:17:30 UTC 2019
280 Sat Mar 16 17:17:40 UTC 2019
308 Sat Mar 16 17:17:50 UTC 2019
307 Sat Mar 16 17:18:00 UTC 2019
282 Sat Mar 16 17:18:10 UTC 2019
302 Sat Mar 16 17:18:20 UTC 2019
289 Sat Mar 16 17:18:30 UTC 2019
256 Sat Mar 16 17:18:40 UTC 2019
246 Sat Mar 16 17:18:50 UTC 2019
236 Sat Mar 16 17:19:00 UTC 2019
284 Sat Mar 16 17:19:10 UTC 2019
246 Sat Mar 16 17:19:20 UTC 2019
263 Sat Mar 16 17:19:30 UTC 2019
228 Sat Mar 16 17:19:40 UTC 2019
243 Sat Mar 16 17:19:50 UTC 2019
245 Sat Mar 16 17:20:00 UTC 2019
243 Sat Mar 16 17:20:10 UTC 2019
235 Sat Mar 16 17:20:20 UTC 2019
227 Sat Mar 16 17:20:30 UTC 2019
214 Sat Mar 16 17:20:40 UTC 2019
230 Sat Mar 16 17:20:50 UTC 2019
228 Sat Mar 16 17:21:00 UTC 2019
212 Sat Mar 16 17:21:10 UTC 2019
223 Sat Mar 16 17:21:20 UTC 2019
216 Sat Mar 16 17:21:30 UTC 2019
225 Sat Mar 16 17:21:40 UTC 2019
303 Sat Mar 16 17:21:50 UTC 2019
280 Sat Mar 16 17:22:00 UTC 2019
278 Sat Mar 16 17:22:10 UTC 2019
289 Sat Mar 16 17:22:20 UTC 2019
254 Sat Mar 16 17:22:30 UTC 2019
305 Sat Mar 16 17:22:40 UTC 2019
280 Sat Mar 16 17:22:50 UTC 2019
279 Sat Mar 16 17:23:00 UTC 2019
264 Sat Mar 16 17:23:10 UTC 2019
270 Sat Mar 16 17:23:20 UTC 2019
275 Sat Mar 16 17:23:30 UTC 2019
260 Sat Mar 16 17:23:40 UTC 2019
262 Sat Mar 16 17:23:50 UTC 2019
259 Sat Mar 16 17:24:00 UTC 2019
260 Sat Mar 16 17:24:10 UTC 2019
268 Sat Mar 16 17:24:20 UTC 2019
244 Sat Mar 16 17:24:30 UTC 2019
221 Sat Mar 16 17:24:40 UTC 2019
254 Sat Mar 16 17:24:50 UTC 2019
228 Sat Mar 16 17:25:00 UTC 2019
239 Sat Mar 16 17:25:10 UTC 2019
223 Sat Mar 16 17:25:20 UTC 2019
238 Sat Mar 16 17:25:30 UTC 2019
257 Sat Mar 16 17:25:40 UTC 2019
232 Sat Mar 16 17:25:50 UTC 2019
212 Sat Mar 16 17:26:00 UTC 2019
218 Sat Mar 16 17:26:10 UTC 2019
244 Sat Mar 16 17:26:20 UTC 2019
267 Sat Mar 16 17:26:30 UTC 2019
301 Sat Mar 16 17:26:40 UTC 2019
305 Sat Mar 16 17:26:50 UTC 2019
266 Sat Mar 16 17:27:00 UTC 2019
283 Sat Mar 16 17:27:10 UTC 2019
265 Sat Mar 16 17:27:20 UTC 2019
299 Sat Mar 16 17:27:30 UTC 2019
296 Sat Mar 16 17:27:40 UTC 2019
270 Sat Mar 16 17:27:50 UTC 2019
288 Sat Mar 16 17:28:00 UTC 2019
235 Sat Mar 16 17:28:10 UTC 2019
227 Sat Mar 16 17:28:20 UTC 2019
260 Sat Mar 16 17:28:30 UTC 2019
236 Sat Mar 16 17:28:40 UTC 2019
235 Sat Mar 16 17:28:50 UTC 2019
223 Sat Mar 16 17:29:00 UTC 2019
264 Sat Mar 16 17:29:10 UTC 2019
248 Sat Mar 16 17:29:20 UTC 2019
274 Sat Mar 16 17:29:30 UTC 2019
248 Sat Mar 16 17:29:40 UTC 2019
226 Sat Mar 16 17:29:50 UTC 2019
249 Sat Mar 16 17:30:00 UTC 2019
247 Sat Mar 16 17:30:10 UTC 2019
244 Sat Mar 16 17:30:20 UTC 2019
240 Sat Mar 16 17:30:30 UTC 2019
238 Sat Mar 16 17:30:40 UTC 2019
240 Sat Mar 16 17:30:50 UTC 2019
223 Sat Mar 16 17:31:00 UTC 2019
226 Sat Mar 16 17:31:10 UTC 2019
234 Sat Mar 16 17:31:20 UTC 2019
226 Sat Mar 16 17:31:30 UTC 2019
360 Sat Mar 16 17:31:40 UTC 2019
252 Sat Mar 16 17:31:50 UTC 2019
270 Sat Mar 16 17:32:00 UTC 2019
294 Sat Mar 16 17:32:10 UTC 2019
309 Sat Mar 16 17:32:20 UTC 2019
280 Sat Mar 16 17:32:30 UTC 2019
288 Sat Mar 16 17:32:40 UTC 2019
296 Sat Mar 16 17:32:50 UTC 2019
278 Sat Mar 16 17:33:00 UTC 2019
270 Sat Mar 16 17:33:10 UTC 2019
268 Sat Mar 16 17:33:20 UTC 2019
252 Sat Mar 16 17:33:30 UTC 2019
269 Sat Mar 16 17:33:40 UTC 2019
240 Sat Mar 16 17:33:50 UTC 2019
264 Sat Mar 16 17:34:00 UTC 2019
252 Sat Mar 16 17:34:10 UTC 2019
267 Sat Mar 16 17:34:20 UTC 2019
258 Sat Mar 16 17:34:30 UTC 2019
275 Sat Mar 16 17:34:40 UTC 2019
214 Sat Mar 16 17:34:50 UTC 2019
230 Sat Mar 16 17:35:00 UTC 2019
251 Sat Mar 16 17:35:10 UTC 2019
219 Sat Mar 16 17:35:20 UTC 2019
248 Sat Mar 16 17:35:30 UTC 2019
236 Sat Mar 16 17:35:40 UTC 2019
227 Sat Mar 16 17:35:50 UTC 2019
243 Sat Mar 16 17:36:00 UTC 2019
219 Sat Mar 16 17:36:10 UTC 2019
229 Sat Mar 16 17:36:20 UTC 2019
269 Sat Mar 16 17:36:30 UTC 2019
314 Sat Mar 16 17:36:40 UTC 2019
289 Sat Mar 16 17:36:50 UTC 2019
251 Sat Mar 16 17:37:00 UTC 2019
271 Sat Mar 16 17:37:10 UTC 2019
330 Sat Mar 16 17:37:20 UTC 2019
304 Sat Mar 16 17:37:30 UTC 2019
267 Sat Mar 16 17:37:40 UTC 2019
264 Sat Mar 16 17:37:50 UTC 2019
254 Sat Mar 16 17:38:00 UTC 2019
270 Sat Mar 16 17:38:10 UTC 2019
236 Sat Mar 16 17:38:20 UTC 2019
258 Sat Mar 16 17:38:30 UTC 2019
264 Sat Mar 16 17:38:40 UTC 2019
230 Sat Mar 16 17:38:50 UTC 2019
226 Sat Mar 16 17:39:00 UTC 2019
234 Sat Mar 16 17:39:10 UTC 2019
257 Sat Mar 16 17:39:20 UTC 2019
227 Sat Mar 16 17:39:30 UTC 2019
243 Sat Mar 16 17:39:40 UTC 2019
254 Sat Mar 16 17:39:50 UTC 2019
241 Sat Mar 16 17:40:00 UTC 2019
253 Sat Mar 16 17:40:10 UTC 2019
235 Sat Mar 16 17:40:20 UTC 2019
255 Sat Mar 16 17:40:30 UTC 2019
260 Sat Mar 16 17:40:40 UTC 2019
242 Sat Mar 16 17:40:50 UTC 2019
233 Sat Mar 16 17:41:00 UTC 2019
229 Sat Mar 16 17:41:10 UTC 2019
229 Sat Mar 16 17:41:20 UTC 2019
309 Sat Mar 16 17:41:30 UTC 2019
284 Sat Mar 16 17:41:40 UTC 2019
286 Sat Mar 16 17:41:50 UTC 2019
244 Sat Mar 16 17:42:00 UTC 2019
275 Sat Mar 16 17:42:10 UTC 2019
277 Sat Mar 16 17:42:20 UTC 2019
276 Sat Mar 16 17:42:30 UTC 2019
279 Sat Mar 16 17:42:40 UTC 2019
267 Sat Mar 16 17:42:50 UTC 2019
284 Sat Mar 16 17:43:00 UTC 2019
275 Sat Mar 16 17:43:10 UTC 2019
261 Sat Mar 16 17:43:20 UTC 2019
269 Sat Mar 16 17:43:30 UTC 2019
243 Sat Mar 16 17:43:40 UTC 2019
246 Sat Mar 16 17:43:50 UTC 2019
259 Sat Mar 16 17:44:00 UTC 2019
285 Sat Mar 16 17:44:10 UTC 2019
258 Sat Mar 16 17:44:20 UTC 2019
271 Sat Mar 16 17:44:30 UTC 2019
251 Sat Mar 16 17:44:40 UTC 2019
257 Sat Mar 16 17:44:50 UTC 2019
234 Sat Mar 16 17:45:00 UTC 2019
235 Sat Mar 16 17:45:10 UTC 2019
212 Sat Mar 16 17:45:20 UTC 2019
249 Sat Mar 16 17:45:30 UTC 2019
244 Sat Mar 16 17:45:40 UTC 2019
228 Sat Mar 16 17:45:50 UTC 2019
244 Sat Mar 16 17:46:00 UTC 2019
258 Sat Mar 16 17:46:10 UTC 2019
305 Sat Mar 16 17:46:20 UTC 2019
279 Sat Mar 16 17:46:30 UTC 2019
294 Sat Mar 16 17:46:40 UTC 2019
261 Sat Mar 16 17:46:50 UTC 2019
283 Sat Mar 16 17:47:00 UTC 2019
302 Sat Mar 16 17:47:10 UTC 2019
292 Sat Mar 16 17:47:20 UTC 2019
274 Sat Mar 16 17:47:30 UTC 2019
289 Sat Mar 16 17:47:40 UTC 2019
256 Sat Mar 16 17:47:50 UTC 2019
278 Sat Mar 16 17:48:00 UTC 2019
261 Sat Mar 16 17:48:10 UTC 2019
236 Sat Mar 16 17:48:20 UTC 2019
241 Sat Mar 16 17:48:30 UTC 2019
242 Sat Mar 16 17:48:40 UTC 2019
266 Sat Mar 16 17:48:50 UTC 2019
254 Sat Mar 16 17:49:00 UTC 2019
241 Sat Mar 16 17:49:10 UTC 2019
260 Sat Mar 16 17:49:20 UTC 2019
229 Sat Mar 16 17:49:30 UTC 2019
246 Sat Mar 16 17:49:40 UTC 2019
232 Sat Mar 16 17:49:50 UTC 2019
231 Sat Mar 16 17:50:00 UTC 2019
200 Sat Mar 16 17:50:10 UTC 2019
246 Sat Mar 16 17:50:20 UTC 2019
259 Sat Mar 16 17:50:30 UTC 2019
249 Sat Mar 16 17:50:40 UTC 2019
246 Sat Mar 16 17:50:50 UTC 2019
242 Sat Mar 16 17:51:00 UTC 2019
229 Sat Mar 16 17:51:10 UTC 2019
305 Sat Mar 16 17:51:20 UTC 2019
281 Sat Mar 16 17:51:30 UTC 2019
285 Sat Mar 16 17:51:40 UTC 2019
250 Sat Mar 16 17:51:50 UTC 2019
284 Sat Mar 16 17:52:00 UTC 2019
296 Sat Mar 16 17:52:10 UTC 2019
279 Sat Mar 16 17:52:20 UTC 2019
280 Sat Mar 16 17:52:30 UTC 2019
277 Sat Mar 16 17:52:40 UTC 2019
289 Sat Mar 16 17:52:50 UTC 2019
261 Sat Mar 16 17:53:00 UTC 2019
257 Sat Mar 16 17:53:10 UTC 2019
266 Sat Mar 16 17:53:20 UTC 2019
241 Sat Mar 16 17:53:30 UTC 2019
238 Sat Mar 16 17:53:40 UTC 2019
219 Sat Mar 16 17:53:50 UTC 2019
287 Sat Mar 16 17:54:00 UTC 2019
254 Sat Mar 16 17:54:10 UTC 2019
251 Sat Mar 16 17:54:20 UTC 2019
245 Sat Mar 16 17:54:30 UTC 2019
238 Sat Mar 16 17:54:40 UTC 2019
229 Sat Mar 16 17:54:50 UTC 2019
234 Sat Mar 16 17:55:00 UTC 2019
218 Sat Mar 16 17:55:10 UTC 2019
247 Sat Mar 16 17:55:20 UTC 2019
229 Sat Mar 16 17:55:30 UTC 2019
249 Sat Mar 16 17:55:40 UTC 2019
226 Sat Mar 16 17:55:50 UTC 2019
259 Sat Mar 16 17:56:00 UTC 2019
235 Sat Mar 16 17:56:10 UTC 2019
302 Sat Mar 16 17:56:20 UTC 2019
284 Sat Mar 16 17:56:30 UTC 2019
294 Sat Mar 16 17:56:40 UTC 2019
276 Sat Mar 16 17:56:50 UTC 2019
316 Sat Mar 16 17:57:00 UTC 2019
301 Sat Mar 16 17:57:10 UTC 2019
299 Sat Mar 16 17:57:20 UTC 2019
303 Sat Mar 16 17:57:30 UTC 2019
287 Sat Mar 16 17:57:40 UTC 2019
263 Sat Mar 16 17:57:50 UTC 2019
233 Sat Mar 16 17:58:00 UTC 2019
253 Sat Mar 16 17:58:10 UTC 2019
267 Sat Mar 16 17:58:20 UTC 2019
234 Sat Mar 16 17:58:30 UTC 2019
258 Sat Mar 16 17:58:40 UTC 2019
254 Sat Mar 16 17:58:50 UTC 2019
250 Sat Mar 16 17:59:00 UTC 2019
232 Sat Mar 16 17:59:10 UTC 2019
285 Sat Mar 16 17:59:20 UTC 2019
253 Sat Mar 16 17:59:30 UTC 2019
244 Sat Mar 16 17:59:40 UTC 2019
256 Sat Mar 16 17:59:50 UTC 2019
227 Sat Mar 16 18:00:00 UTC 2019
221 Sat Mar 16 18:00:10 UTC 2019
231 Sat Mar 16 18:00:20 UTC 2019
251 Sat Mar 16 18:00:30 UTC 2019
236 Sat Mar 16 18:00:40 UTC 2019
244 Sat Mar 16 18:00:50 UTC 2019
221 Sat Mar 16 18:01:00 UTC 2019
248 Sat Mar 16 18:01:10 UTC 2019
302 Sat Mar 16 18:01:20 UTC 2019
280 Sat Mar 16 18:01:30 UTC 2019
301 Sat Mar 16 18:01:40 UTC 2019
293 Sat Mar 16 18:01:50 UTC 2019
303 Sat Mar 16 18:02:00 UTC 2019
299 Sat Mar 16 18:02:10 UTC 2019
293 Sat Mar 16 18:02:20 UTC 2019
303 Sat Mar 16 18:02:30 UTC 2019
261 Sat Mar 16 18:02:40 UTC 2019
270 Sat Mar 16 18:02:50 UTC 2019
255 Sat Mar 16 18:03:00 UTC 2019
239 Sat Mar 16 18:03:10 UTC 2019
269 Sat Mar 16 18:03:20 UTC 2019
251 Sat Mar 16 18:03:30 UTC 2019
267 Sat Mar 16 18:03:40 UTC 2019
260 Sat Mar 16 18:03:50 UTC 2019
264 Sat Mar 16 18:04:00 UTC 2019
263 Sat Mar 16 18:04:10 UTC 2019
256 Sat Mar 16 18:04:20 UTC 2019
279 Sat Mar 16 18:04:30 UTC 2019
226 Sat Mar 16 18:04:40 UTC 2019
229 Sat Mar 16 18:04:50 UTC 2019
256 Sat Mar 16 18:05:00 UTC 2019
220 Sat Mar 16 18:05:10 UTC 2019
249 Sat Mar 16 18:05:20 UTC 2019
233 Sat Mar 16 18:05:30 UTC 2019
239 Sat Mar 16 18:05:40 UTC 2019
244 Sat Mar 16 18:05:50 UTC 2019
226 Sat Mar 16 18:06:00 UTC 2019
270 Sat Mar 16 18:06:10 UTC 2019
305 Sat Mar 16 18:06:20 UTC 2019
292 Sat Mar 16 18:06:30 UTC 2019
246 Sat Mar 16 18:06:40 UTC 2019
318 Sat Mar 16 18:06:50 UTC 2019
287 Sat Mar 16 18:07:00 UTC 2019
257 Sat Mar 16 18:07:10 UTC 2019
269 Sat Mar 16 18:07:20 UTC 2019
267 Sat Mar 16 18:07:30 UTC 2019
290 Sat Mar 16 18:07:40 UTC 2019
269 Sat Mar 16 18:07:50 UTC 2019
258 Sat Mar 16 18:08:00 UTC 2019
268 Sat Mar 16 18:08:10 UTC 2019
259 Sat Mar 16 18:08:20 UTC 2019
243 Sat Mar 16 18:08:30 UTC 2019
217 Sat Mar 16 18:08:40 UTC 2019
217 Sat Mar 16 18:08:50 UTC 2019
266 Sat Mar 16 18:09:00 UTC 2019
271 Sat Mar 16 18:09:10 UTC 2019
247 Sat Mar 16 18:09:20 UTC 2019
245 Sat Mar 16 18:09:30 UTC 2019
225 Sat Mar 16 18:09:40 UTC 2019
258 Sat Mar 16 18:09:50 UTC 2019
225 Sat Mar 16 18:10:00 UTC 2019
241 Sat Mar 16 18:10:10 UTC 2019
243 Sat Mar 16 18:10:20 UTC 2019
252 Sat Mar 16 18:10:30 UTC 2019
255 Sat Mar 16 18:10:40 UTC 2019
246 Sat Mar 16 18:10:50 UTC 2019
257 Sat Mar 16 18:11:00 UTC 2019
289 Sat Mar 16 18:11:10 UTC 2019
296 Sat Mar 16 18:11:20 UTC 2019
280 Sat Mar 16 18:11:30 UTC 2019
294 Sat Mar 16 18:11:40 UTC 2019
289 Sat Mar 16 18:11:50 UTC 2019
306 Sat Mar 16 18:12:00 UTC 2019
259 Sat Mar 16 18:12:10 UTC 2019
300 Sat Mar 16 18:12:20 UTC 2019
300 Sat Mar 16 18:12:30 UTC 2019
270 Sat Mar 16 18:12:40 UTC 2019
289 Sat Mar 16 18:12:50 UTC 2019
294 Sat Mar 16 18:13:00 UTC 2019
259 Sat Mar 16 18:13:10 UTC 2019
247 Sat Mar 16 18:13:20 UTC 2019
264 Sat Mar 16 18:13:30 UTC 2019
258 Sat Mar 16 18:13:40 UTC 2019
251 Sat Mar 16 18:13:50 UTC 2019
261 Sat Mar 16 18:14:00 UTC 2019
247 Sat Mar 16 18:14:10 UTC 2019
247 Sat Mar 16 18:14:20 UTC 2019
214 Sat Mar 16 18:14:30 UTC 2019
282 Sat Mar 16 18:14:40 UTC 2019
230 Sat Mar 16 18:14:50 UTC 2019
246 Sat Mar 16 18:15:00 UTC 2019
205 Sat Mar 16 18:15:10 UTC 2019
243 Sat Mar 16 18:15:20 UTC 2019
241 Sat Mar 16 18:15:30 UTC 2019
236 Sat Mar 16 18:15:40 UTC 2019
295 Sat Mar 16 18:15:50 UTC 2019
271 Sat Mar 16 18:16:00 UTC 2019
275 Sat Mar 16 18:16:10 UTC 2019
275 Sat Mar 16 18:16:20 UTC 2019
272 Sat Mar 16 18:16:30 UTC 2019
278 Sat Mar 16 18:16:40 UTC 2019
293 Sat Mar 16 18:16:50 UTC 2019
290 Sat Mar 16 18:17:00 UTC 2019
268 Sat Mar 16 18:17:10 UTC 2019
262 Sat Mar 16 18:17:20 UTC 2019
252 Sat Mar 16 18:17:30 UTC 2019
238 Sat Mar 16 18:17:40 UTC 2019
254 Sat Mar 16 18:17:50 UTC 2019
254 Sat Mar 16 18:18:00 UTC 2019
262 Sat Mar 16 18:18:10 UTC 2019
232 Sat Mar 16 18:18:20 UTC 2019
225 Sat Mar 16 18:18:30 UTC 2019
240 Sat Mar 16 18:18:40 UTC 2019
240 Sat Mar 16 18:18:50 UTC 2019
259 Sat Mar 16 18:19:00 UTC 2019
240 Sat Mar 16 18:19:10 UTC 2019
250 Sat Mar 16 18:19:20 UTC 2019
271 Sat Mar 16 18:19:30 UTC 2019
248 Sat Mar 16 18:19:40 UTC 2019
206 Sat Mar 16 18:19:50 UTC 2019
223 Sat Mar 16 18:20:00 UTC 2019
233 Sat Mar 16 18:20:10 UTC 2019
246 Sat Mar 16 18:20:20 UTC 2019
232 Sat Mar 16 18:20:30 UTC 2019
281 Sat Mar 16 18:20:40 UTC 2019
333 Sat Mar 16 18:20:50 UTC 2019
297 Sat Mar 16 18:21:00 UTC 2019
294 Sat Mar 16 18:21:10 UTC 2019
258 Sat Mar 16 18:21:20 UTC 2019
305 Sat Mar 16 18:21:30 UTC 2019
331 Sat Mar 16 18:21:40 UTC 2019
268 Sat Mar 16 18:21:50 UTC 2019
310 Sat Mar 16 18:22:00 UTC 2019
274 Sat Mar 16 18:22:10 UTC 2019
271 Sat Mar 16 18:22:20 UTC 2019
260 Sat Mar 16 18:22:30 UTC 2019
284 Sat Mar 16 18:22:40 UTC 2019
268 Sat Mar 16 18:22:50 UTC 2019
244 Sat Mar 16 18:23:00 UTC 2019
237 Sat Mar 16 18:23:10 UTC 2019
261 Sat Mar 16 18:23:20 UTC 2019
261 Sat Mar 16 18:23:30 UTC 2019
248 Sat Mar 16 18:23:40 UTC 2019
250 Sat Mar 16 18:23:50 UTC 2019
274 Sat Mar 16 18:24:00 UTC 2019
255 Sat Mar 16 18:24:10 UTC 2019
259 Sat Mar 16 18:24:20 UTC 2019
242 Sat Mar 16 18:24:30 UTC 2019
261 Sat Mar 16 18:24:40 UTC 2019
249 Sat Mar 16 18:24:50 UTC 2019
253 Sat Mar 16 18:25:00 UTC 2019
246 Sat Mar 16 18:25:10 UTC 2019
260 Sat Mar 16 18:25:20 UTC 2019
296 Sat Mar 16 18:25:30 UTC 2019
283 Sat Mar 16 18:25:40 UTC 2019
281 Sat Mar 16 18:25:50 UTC 2019
275 Sat Mar 16 18:26:00 UTC 2019
292 Sat Mar 16 18:26:10 UTC 2019
345 Sat Mar 16 18:26:20 UTC 2019
271 Sat Mar 16 18:26:30 UTC 2019
305 Sat Mar 16 18:26:40 UTC 2019
298 Sat Mar 16 18:26:50 UTC 2019
292 Sat Mar 16 18:27:00 UTC 2019
305 Sat Mar 16 18:27:10 UTC 2019
254 Sat Mar 16 18:27:20 UTC 2019
275 Sat Mar 16 18:27:30 UTC 2019
255 Sat Mar 16 18:27:40 UTC 2019
260 Sat Mar 16 18:27:50 UTC 2019
252 Sat Mar 16 18:28:00 UTC 2019
249 Sat Mar 16 18:28:10 UTC 2019
272 Sat Mar 16 18:28:20 UTC 2019
253 Sat Mar 16 18:28:30 UTC 2019
254 Sat Mar 16 18:28:40 UTC 2019
262 Sat Mar 16 18:28:50 UTC 2019
228 Sat Mar 16 18:29:00 UTC 2019
258 Sat Mar 16 18:29:10 UTC 2019
231 Sat Mar 16 18:29:20 UTC 2019
267 Sat Mar 16 18:29:30 UTC 2019
216 Sat Mar 16 18:29:40 UTC 2019
281 Sat Mar 16 18:29:50 UTC 2019
236 Sat Mar 16 18:30:00 UTC 2019
270 Sat Mar 16 18:30:10 UTC 2019
262 Sat Mar 16 18:30:20 UTC 2019
299 Sat Mar 16 18:30:30 UTC 2019
283 Sat Mar 16 18:30:40 UTC 2019
299 Sat Mar 16 18:30:50 UTC 2019
293 Sat Mar 16 18:31:00 UTC 2019
270 Sat Mar 16 18:31:10 UTC 2019
344 Sat Mar 16 18:31:20 UTC 2019
299 Sat Mar 16 18:31:30 UTC 2019
291 Sat Mar 16 18:31:40 UTC 2019
250 Sat Mar 16 18:31:50 UTC 2019
262 Sat Mar 16 18:32:00 UTC 2019
268 Sat Mar 16 18:32:10 UTC 2019
293 Sat Mar 16 18:32:20 UTC 2019
273 Sat Mar 16 18:32:30 UTC 2019
278 Sat Mar 16 18:32:40 UTC 2019
277 Sat Mar 16 18:32:50 UTC 2019
231 Sat Mar 16 18:33:00 UTC 2019
282 Sat Mar 16 18:33:10 UTC 2019
265 Sat Mar 16 18:33:20 UTC 2019
258 Sat Mar 16 18:33:30 UTC 2019
227 Sat Mar 16 18:33:40 UTC 2019
219 Sat Mar 16 18:33:50 UTC 2019
267 Sat Mar 16 18:34:00 UTC 2019
247 Sat Mar 16 18:34:10 UTC 2019
249 Sat Mar 16 18:34:20 UTC 2019
252 Sat Mar 16 18:34:30 UTC 2019
278 Sat Mar 16 18:34:40 UTC 2019
264 Sat Mar 16 18:34:50 UTC 2019
225 Sat Mar 16 18:35:00 UTC 2019
238 Sat Mar 16 18:35:10 UTC 2019
355 Sat Mar 16 18:35:20 UTC 2019
336 Sat Mar 16 18:35:30 UTC 2019
310 Sat Mar 16 18:35:40 UTC 2019
292 Sat Mar 16 18:35:50 UTC 2019
271 Sat Mar 16 18:36:00 UTC 2019
340 Sat Mar 16 18:36:10 UTC 2019
309 Sat Mar 16 18:36:20 UTC 2019
296 Sat Mar 16 18:36:30 UTC 2019
305 Sat Mar 16 18:36:40 UTC 2019
291 Sat Mar 16 18:36:50 UTC 2019
320 Sat Mar 16 18:37:00 UTC 2019
271 Sat Mar 16 18:37:10 UTC 2019
263 Sat Mar 16 18:37:20 UTC 2019
264 Sat Mar 16 18:37:30 UTC 2019
294 Sat Mar 16 18:37:40 UTC 2019
272 Sat Mar 16 18:37:50 UTC 2019
258 Sat Mar 16 18:38:00 UTC 2019
271 Sat Mar 16 18:38:10 UTC 2019
235 Sat Mar 16 18:38:20 UTC 2019
221 Sat Mar 16 18:38:30 UTC 2019
242 Sat Mar 16 18:38:40 UTC 2019
262 Sat Mar 16 18:38:50 UTC 2019
288 Sat Mar 16 18:39:00 UTC 2019
238 Sat Mar 16 18:39:10 UTC 2019
238 Sat Mar 16 18:39:20 UTC 2019
236 Sat Mar 16 18:39:30 UTC 2019
273 Sat Mar 16 18:39:40 UTC 2019
232 Sat Mar 16 18:39:50 UTC 2019
259 Sat Mar 16 18:40:00 UTC 2019
337 Sat Mar 16 18:40:10 UTC 2019
307 Sat Mar 16 18:40:20 UTC 2019
294 Sat Mar 16 18:40:30 UTC 2019
242 Sat Mar 16 18:40:40 UTC 2019
296 Sat Mar 16 18:40:50 UTC 2019
282 Sat Mar 16 18:41:00 UTC 2019
326 Sat Mar 16 18:41:10 UTC 2019
296 Sat Mar 16 18:41:20 UTC 2019
315 Sat Mar 16 18:41:30 UTC 2019
319 Sat Mar 16 18:41:40 UTC 2019
300 Sat Mar 16 18:41:50 UTC 2019
293 Sat Mar 16 18:42:00 UTC 2019
307 Sat Mar 16 18:42:10 UTC 2019
267 Sat Mar 16 18:42:20 UTC 2019
247 Sat Mar 16 18:42:30 UTC 2019
237 Sat Mar 16 18:42:40 UTC 2019
286 Sat Mar 16 18:42:50 UTC 2019
234 Sat Mar 16 18:43:00 UTC 2019
267 Sat Mar 16 18:43:10 UTC 2019
241 Sat Mar 16 18:43:20 UTC 2019
250 Sat Mar 16 18:43:30 UTC 2019
255 Sat Mar 16 18:43:40 UTC 2019
245 Sat Mar 16 18:43:50 UTC 2019
259 Sat Mar 16 18:44:00 UTC 2019
267 Sat Mar 16 18:44:10 UTC 2019
271 Sat Mar 16 18:44:20 UTC 2019
227 Sat Mar 16 18:44:30 UTC 2019
290 Sat Mar 16 18:44:40 UTC 2019
245 Sat Mar 16 18:44:50 UTC 2019
376 Sat Mar 16 18:45:00 UTC 2019
320 Sat Mar 16 18:45:10 UTC 2019
313 Sat Mar 16 18:45:20 UTC 2019
297 Sat Mar 16 18:45:30 UTC 2019
Grouped by 100s
2567	Mar 16 2019 16:38:20
2450	Mar 16 2019 16:40:00
2631	Mar 16 2019 16:41:40
2503	Mar 16 2019 16:43:20
2265	Mar 16 2019 16:45:00
2622	Mar 16 2019 16:46:40
2514	Mar 16 2019 16:48:20
2302	Mar 16 2019 16:50:00
2635	Mar 16 2019 16:51:40
2487	Mar 16 2019 16:53:20
2248	Mar 16 2019 16:55:00
2602	Mar 16 2019 16:56:40
2519	Mar 16 2019 16:58:20
2345	Mar 16 2019 17:00:00
2641	Mar 16 2019 17:01:40
2604	Mar 16 2019 17:03:20
2376	Mar 16 2019 17:05:00
2747	Mar 16 2019 17:06:40
2629	Mar 16 2019 17:08:20
2434	Mar 16 2019 17:10:00
2718	Mar 16 2019 17:11:40
2649	Mar 16 2019 17:13:20
2440	Mar 16 2019 17:15:00
2745	Mar 16 2019 17:16:40
2593	Mar 16 2019 17:18:20
2273	Mar 16 2019 17:20:00
2757	Mar 16 2019 17:21:40
2573	Mar 16 2019 17:23:20
2358	Mar 16 2019 17:25:00
2808	Mar 16 2019 17:26:40
2441	Mar 16 2019 17:28:20
2367	Mar 16 2019 17:30:00
2897	Mar 16 2019 17:31:40
2559	Mar 16 2019 17:33:20
2371	Mar 16 2019 17:35:00
2814	Mar 16 2019 17:36:40
2429	Mar 16 2019 17:38:20
2486	Mar 16 2019 17:40:00
2747	Mar 16 2019 17:41:40
2600	Mar 16 2019 17:43:20
2488	Mar 16 2019 17:45:00
2790	Mar 16 2019 17:46:40
2447	Mar 16 2019 17:48:20
2488	Mar 16 2019 17:50:00
2758	Mar 16 2019 17:51:40
2468	Mar 16 2019 17:53:20
2483	Mar 16 2019 17:55:00
2825	Mar 16 2019 17:56:40
2533	Mar 16 2019 17:58:20
2461	Mar 16 2019 18:00:00
2817	Mar 16 2019 18:01:40
2564	Mar 16 2019 18:03:20
2534	Mar 16 2019 18:05:00
2729	Mar 16 2019 18:06:40
2448	Mar 16 2019 18:08:20
2584	Mar 16 2019 18:10:00
2860	Mar 16 2019 18:11:40
2501	Mar 16 2019 18:13:20
2559	Mar 16 2019 18:15:00
2651	Mar 16 2019 18:16:40
2411	Mar 16 2019 18:18:20
2702	Mar 16 2019 18:20:00
2747	Mar 16 2019 18:21:40
2560	Mar 16 2019 18:23:20
2802	Mar 16 2019 18:25:00
2745	Mar 16 2019 18:26:40
2522	Mar 16 2019 18:28:20
2855	Mar 16 2019 18:30:00
2705	Mar 16 2019 18:31:40
2526	Mar 16 2019 18:33:20
2972	Mar 16 2019 18:35:00
2809	Mar 16 2019 18:36:40
2465	Mar 16 2019 18:38:20
2954	Mar 16 2019 18:40:00
2757	Mar 16 2019 18:41:40
2550	Mar 16 2019 18:43:20

And plotted:

100s-rate.png (820×2 px, 260 KB)

Like in 2017, we've still got the regular flow with peaks and dips following a 5-min cadence.

10s-rate-eqiad.png (800×2 px, 230 KB)

I also added a 10s rate above to see how closely aligned they really are and it shows that while naturally a bit less smooth, the 5-min window is still quite visible there as well.

Change 496497 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: set /w/load.php Age to 0

https://gerrit.wikimedia.org/r/496497

Change 496497 merged by Ema:
[operations/puppet@production] varnish: set /w/load.php Age to 0

https://gerrit.wikimedia.org/r/496497

Krinkle closed this task as Resolved.EditedMar 26 2019, 7:41 PM
Krinkle edited projects, added Wikimedia-Performance-publish; removed Patch-For-Review.
Startup request rate pattern

And server-side we can confirm the distribution of the startup request (which we changed) is now much more even (compared to T105657#5029880).

Grouped by 100s
3778 Tue Mar 26 17:28:20 UTC 2019
3751 Tue Mar 26 17:30:00 UTC 2019
3768 Tue Mar 26 17:31:40 UTC 2019
3801 Tue Mar 26 17:33:20 UTC 2019
3796 Tue Mar 26 17:35:00 UTC 2019
3972 Tue Mar 26 17:36:40 UTC 2019
3794 Tue Mar 26 17:38:20 UTC 2019
3936 Tue Mar 26 17:40:00 UTC 2019
3870 Tue Mar 26 17:41:40 UTC 2019
3917 Tue Mar 26 17:43:20 UTC 2019
4011 Tue Mar 26 17:45:00 UTC 2019
4012 Tue Mar 26 17:46:40 UTC 2019
3903 Tue Mar 26 17:48:20 UTC 2019
4045 Tue Mar 26 17:50:00 UTC 2019
3914 Tue Mar 26 17:51:40 UTC 2019
3979 Tue Mar 26 17:53:20 UTC 2019
4017 Tue Mar 26 17:55:00 UTC 2019
3937 Tue Mar 26 17:56:40 UTC 2019
4057 Tue Mar 26 17:58:20 UTC 2019
3857 Tue Mar 26 18:00:00 UTC 2019
3789 Tue Mar 26 18:01:40 UTC 2019
3966 Tue Mar 26 18:03:20 UTC 2019
3750 Tue Mar 26 18:05:00 UTC 2019
3741 Tue Mar 26 18:06:40 UTC 2019
3912 Tue Mar 26 18:08:20 UTC 2019
3927 Tue Mar 26 18:10:00 UTC 2019
3921 Tue Mar 26 18:11:40 UTC 2019
3835 Tue Mar 26 18:13:20 UTC 2019
3917 Tue Mar 26 18:15:00 UTC 2019
3855 Tue Mar 26 18:16:40 UTC 2019
3954 Tue Mar 26 18:18:20 UTC 2019
3814 Tue Mar 26 18:20:00 UTC 2019
3841 Tue Mar 26 18:21:40 UTC 2019
3911 Tue Mar 26 18:23:20 UTC 2019
3935 Tue Mar 26 18:25:00 UTC 2019
3915 Tue Mar 26 18:26:40 UTC 2019
3882 Tue Mar 26 18:28:20 UTC 2019
3870 Tue Mar 26 18:30:00 UTC 2019
3852 Tue Mar 26 18:31:40 UTC 2019
4021 Tue Mar 26 18:33:20 UTC 2019
3976 Tue Mar 26 18:35:00 UTC 2019
3941 Tue Mar 26 18:36:40 UTC 2019
3872 Tue Mar 26 18:38:20 UTC 2019
3932 Tue Mar 26 18:40:00 UTC 2019
3882 Tue Mar 26 18:41:40 UTC 2019
4117 Tue Mar 26 18:43:20 UTC 2019
3966 Tue Mar 26 18:45:00 UTC 2019
3901 Tue Mar 26 18:46:40 UTC 2019
3861 Tue Mar 26 18:48:20 UTC 2019
3950 Tue Mar 26 18:50:00 UTC 2019
Grouped by 10s
361 Tue Mar 26 17:27:10 UTC 2019
390 Tue Mar 26 17:27:20 UTC 2019
350 Tue Mar 26 17:27:30 UTC 2019
376 Tue Mar 26 17:27:40 UTC 2019
380 Tue Mar 26 17:27:50 UTC 2019
361 Tue Mar 26 17:28:00 UTC 2019
389 Tue Mar 26 17:28:10 UTC 2019
331 Tue Mar 26 17:28:20 UTC 2019
369 Tue Mar 26 17:28:30 UTC 2019
387 Tue Mar 26 17:28:40 UTC 2019
375 Tue Mar 26 17:28:50 UTC 2019
365 Tue Mar 26 17:29:00 UTC 2019
370 Tue Mar 26 17:29:10 UTC 2019
387 Tue Mar 26 17:29:20 UTC 2019
408 Tue Mar 26 17:29:30 UTC 2019
379 Tue Mar 26 17:29:40 UTC 2019
407 Tue Mar 26 17:29:50 UTC 2019
407 Tue Mar 26 17:30:00 UTC 2019
381 Tue Mar 26 17:30:10 UTC 2019
370 Tue Mar 26 17:30:20 UTC 2019
408 Tue Mar 26 17:30:30 UTC 2019
391 Tue Mar 26 17:30:40 UTC 2019
329 Tue Mar 26 17:30:50 UTC 2019
380 Tue Mar 26 17:31:00 UTC 2019
356 Tue Mar 26 17:31:10 UTC 2019
368 Tue Mar 26 17:31:20 UTC 2019
361 Tue Mar 26 17:31:30 UTC 2019
401 Tue Mar 26 17:31:40 UTC 2019
406 Tue Mar 26 17:31:50 UTC 2019
364 Tue Mar 26 17:32:00 UTC 2019
391 Tue Mar 26 17:32:10 UTC 2019
323 Tue Mar 26 17:32:20 UTC 2019
389 Tue Mar 26 17:32:30 UTC 2019
378 Tue Mar 26 17:32:40 UTC 2019
387 Tue Mar 26 17:32:50 UTC 2019
369 Tue Mar 26 17:33:00 UTC 2019
360 Tue Mar 26 17:33:10 UTC 2019
372 Tue Mar 26 17:33:20 UTC 2019
396 Tue Mar 26 17:33:30 UTC 2019
354 Tue Mar 26 17:33:40 UTC 2019
362 Tue Mar 26 17:33:50 UTC 2019
405 Tue Mar 26 17:34:00 UTC 2019
380 Tue Mar 26 17:34:10 UTC 2019
378 Tue Mar 26 17:34:20 UTC 2019
378 Tue Mar 26 17:34:30 UTC 2019
370 Tue Mar 26 17:34:40 UTC 2019
406 Tue Mar 26 17:34:50 UTC 2019
349 Tue Mar 26 17:35:00 UTC 2019
408 Tue Mar 26 17:35:10 UTC 2019
398 Tue Mar 26 17:35:20 UTC 2019
384 Tue Mar 26 17:35:30 UTC 2019
382 Tue Mar 26 17:35:40 UTC 2019
367 Tue Mar 26 17:35:50 UTC 2019
367 Tue Mar 26 17:36:00 UTC 2019
384 Tue Mar 26 17:36:10 UTC 2019
386 Tue Mar 26 17:36:20 UTC 2019
371 Tue Mar 26 17:36:30 UTC 2019
385 Tue Mar 26 17:36:40 UTC 2019
386 Tue Mar 26 17:36:50 UTC 2019
409 Tue Mar 26 17:37:00 UTC 2019
409 Tue Mar 26 17:37:10 UTC 2019
444 Tue Mar 26 17:37:20 UTC 2019
379 Tue Mar 26 17:37:30 UTC 2019
413 Tue Mar 26 17:37:40 UTC 2019
381 Tue Mar 26 17:37:50 UTC 2019
388 Tue Mar 26 17:38:00 UTC 2019
378 Tue Mar 26 17:38:10 UTC 2019
365 Tue Mar 26 17:38:20 UTC 2019
348 Tue Mar 26 17:38:30 UTC 2019
423 Tue Mar 26 17:38:40 UTC 2019
422 Tue Mar 26 17:38:50 UTC 2019
367 Tue Mar 26 17:39:00 UTC 2019
361 Tue Mar 26 17:39:10 UTC 2019
374 Tue Mar 26 17:39:20 UTC 2019
381 Tue Mar 26 17:39:30 UTC 2019
376 Tue Mar 26 17:39:40 UTC 2019
377 Tue Mar 26 17:39:50 UTC 2019
404 Tue Mar 26 17:40:00 UTC 2019
407 Tue Mar 26 17:40:10 UTC 2019
412 Tue Mar 26 17:40:20 UTC 2019
383 Tue Mar 26 17:40:30 UTC 2019
408 Tue Mar 26 17:40:40 UTC 2019
366 Tue Mar 26 17:40:50 UTC 2019
390 Tue Mar 26 17:41:00 UTC 2019
369 Tue Mar 26 17:41:10 UTC 2019
397 Tue Mar 26 17:41:20 UTC 2019
400 Tue Mar 26 17:41:30 UTC 2019
399 Tue Mar 26 17:41:40 UTC 2019
384 Tue Mar 26 17:41:50 UTC 2019
383 Tue Mar 26 17:42:00 UTC 2019
379 Tue Mar 26 17:42:10 UTC 2019
388 Tue Mar 26 17:42:20 UTC 2019
425 Tue Mar 26 17:42:30 UTC 2019
385 Tue Mar 26 17:42:40 UTC 2019
365 Tue Mar 26 17:42:50 UTC 2019
399 Tue Mar 26 17:43:00 UTC 2019
363 Tue Mar 26 17:43:10 UTC 2019
412 Tue Mar 26 17:43:20 UTC 2019
406 Tue Mar 26 17:43:30 UTC 2019
399 Tue Mar 26 17:43:40 UTC 2019
381 Tue Mar 26 17:43:50 UTC 2019
367 Tue Mar 26 17:44:00 UTC 2019
388 Tue Mar 26 17:44:10 UTC 2019
395 Tue Mar 26 17:44:20 UTC 2019
401 Tue Mar 26 17:44:30 UTC 2019
403 Tue Mar 26 17:44:40 UTC 2019
365 Tue Mar 26 17:44:50 UTC 2019
439 Tue Mar 26 17:45:00 UTC 2019
366 Tue Mar 26 17:45:10 UTC 2019
386 Tue Mar 26 17:45:20 UTC 2019
383 Tue Mar 26 17:45:30 UTC 2019
408 Tue Mar 26 17:45:40 UTC 2019
414 Tue Mar 26 17:45:50 UTC 2019
368 Tue Mar 26 17:46:00 UTC 2019
417 Tue Mar 26 17:46:10 UTC 2019
435 Tue Mar 26 17:46:20 UTC 2019
395 Tue Mar 26 17:46:30 UTC 2019
455 Tue Mar 26 17:46:40 UTC 2019
391 Tue Mar 26 17:46:50 UTC 2019
422 Tue Mar 26 17:47:00 UTC 2019
409 Tue Mar 26 17:47:10 UTC 2019
402 Tue Mar 26 17:47:20 UTC 2019
387 Tue Mar 26 17:47:30 UTC 2019
391 Tue Mar 26 17:47:40 UTC 2019
389 Tue Mar 26 17:47:50 UTC 2019
397 Tue Mar 26 17:48:00 UTC 2019
369 Tue Mar 26 17:48:10 UTC 2019
410 Tue Mar 26 17:48:20 UTC 2019
410 Tue Mar 26 17:48:30 UTC 2019
390 Tue Mar 26 17:48:40 UTC 2019
388 Tue Mar 26 17:48:50 UTC 2019
368 Tue Mar 26 17:49:00 UTC 2019
392 Tue Mar 26 17:49:10 UTC 2019
412 Tue Mar 26 17:49:20 UTC 2019
391 Tue Mar 26 17:49:30 UTC 2019
367 Tue Mar 26 17:49:40 UTC 2019
375 Tue Mar 26 17:49:50 UTC 2019
394 Tue Mar 26 17:50:00 UTC 2019
374 Tue Mar 26 17:50:10 UTC 2019
385 Tue Mar 26 17:50:20 UTC 2019
415 Tue Mar 26 17:50:30 UTC 2019
390 Tue Mar 26 17:50:40 UTC 2019
406 Tue Mar 26 17:50:50 UTC 2019
447 Tue Mar 26 17:51:00 UTC 2019
405 Tue Mar 26 17:51:10 UTC 2019
362 Tue Mar 26 17:51:20 UTC 2019
467 Tue Mar 26 17:51:30 UTC 2019
395 Tue Mar 26 17:51:40 UTC 2019
388 Tue Mar 26 17:51:50 UTC 2019
384 Tue Mar 26 17:52:00 UTC 2019
393 Tue Mar 26 17:52:10 UTC 2019
395 Tue Mar 26 17:52:20 UTC 2019
370 Tue Mar 26 17:52:30 UTC 2019
377 Tue Mar 26 17:52:40 UTC 2019
416 Tue Mar 26 17:52:50 UTC 2019
404 Tue Mar 26 17:53:00 UTC 2019
392 Tue Mar 26 17:53:10 UTC 2019
368 Tue Mar 26 17:53:20 UTC 2019
412 Tue Mar 26 17:53:30 UTC 2019
393 Tue Mar 26 17:53:40 UTC 2019
398 Tue Mar 26 17:53:50 UTC 2019
365 Tue Mar 26 17:54:00 UTC 2019
428 Tue Mar 26 17:54:10 UTC 2019
417 Tue Mar 26 17:54:20 UTC 2019
373 Tue Mar 26 17:54:30 UTC 2019
427 Tue Mar 26 17:54:40 UTC 2019
398 Tue Mar 26 17:54:50 UTC 2019
413 Tue Mar 26 17:55:00 UTC 2019
405 Tue Mar 26 17:55:10 UTC 2019
405 Tue Mar 26 17:55:20 UTC 2019
378 Tue Mar 26 17:55:30 UTC 2019
439 Tue Mar 26 17:55:40 UTC 2019
407 Tue Mar 26 17:55:50 UTC 2019
381 Tue Mar 26 17:56:00 UTC 2019
396 Tue Mar 26 17:56:10 UTC 2019
393 Tue Mar 26 17:56:20 UTC 2019
400 Tue Mar 26 17:56:30 UTC 2019
371 Tue Mar 26 17:56:40 UTC 2019
341 Tue Mar 26 17:56:50 UTC 2019
390 Tue Mar 26 17:57:00 UTC 2019
408 Tue Mar 26 17:57:10 UTC 2019
379 Tue Mar 26 17:57:20 UTC 2019
429 Tue Mar 26 17:57:30 UTC 2019
403 Tue Mar 26 17:57:40 UTC 2019
398 Tue Mar 26 17:57:50 UTC 2019
417 Tue Mar 26 17:58:00 UTC 2019
401 Tue Mar 26 17:58:10 UTC 2019
394 Tue Mar 26 17:58:20 UTC 2019
382 Tue Mar 26 17:58:30 UTC 2019
426 Tue Mar 26 17:58:40 UTC 2019
420 Tue Mar 26 17:58:50 UTC 2019
391 Tue Mar 26 17:59:00 UTC 2019
443 Tue Mar 26 17:59:10 UTC 2019
381 Tue Mar 26 17:59:20 UTC 2019
431 Tue Mar 26 17:59:30 UTC 2019
390 Tue Mar 26 17:59:40 UTC 2019
399 Tue Mar 26 17:59:50 UTC 2019
394 Tue Mar 26 18:00:00 UTC 2019
368 Tue Mar 26 18:00:10 UTC 2019
417 Tue Mar 26 18:00:20 UTC 2019
386 Tue Mar 26 18:00:30 UTC 2019
367 Tue Mar 26 18:00:40 UTC 2019
401 Tue Mar 26 18:00:50 UTC 2019
396 Tue Mar 26 18:01:00 UTC 2019
374 Tue Mar 26 18:01:10 UTC 2019
364 Tue Mar 26 18:01:20 UTC 2019
390 Tue Mar 26 18:01:30 UTC 2019
384 Tue Mar 26 18:01:40 UTC 2019
382 Tue Mar 26 18:01:50 UTC 2019
384 Tue Mar 26 18:02:00 UTC 2019
435 Tue Mar 26 18:02:10 UTC 2019
349 Tue Mar 26 18:02:20 UTC 2019
367 Tue Mar 26 18:02:30 UTC 2019
384 Tue Mar 26 18:02:40 UTC 2019
373 Tue Mar 26 18:02:50 UTC 2019
383 Tue Mar 26 18:03:00 UTC 2019
348 Tue Mar 26 18:03:10 UTC 2019
399 Tue Mar 26 18:03:20 UTC 2019
425 Tue Mar 26 18:03:30 UTC 2019
392 Tue Mar 26 18:03:40 UTC 2019
385 Tue Mar 26 18:03:50 UTC 2019
416 Tue Mar 26 18:04:00 UTC 2019
387 Tue Mar 26 18:04:10 UTC 2019
414 Tue Mar 26 18:04:20 UTC 2019
391 Tue Mar 26 18:04:30 UTC 2019
400 Tue Mar 26 18:04:40 UTC 2019
357 Tue Mar 26 18:04:50 UTC 2019
399 Tue Mar 26 18:05:00 UTC 2019
370 Tue Mar 26 18:05:10 UTC 2019
394 Tue Mar 26 18:05:20 UTC 2019
366 Tue Mar 26 18:05:30 UTC 2019
370 Tue Mar 26 18:05:40 UTC 2019
362 Tue Mar 26 18:05:50 UTC 2019
345 Tue Mar 26 18:06:00 UTC 2019
393 Tue Mar 26 18:06:10 UTC 2019
375 Tue Mar 26 18:06:20 UTC 2019
376 Tue Mar 26 18:06:30 UTC 2019
342 Tue Mar 26 18:06:40 UTC 2019
388 Tue Mar 26 18:06:50 UTC 2019
365 Tue Mar 26 18:07:00 UTC 2019
380 Tue Mar 26 18:07:10 UTC 2019
393 Tue Mar 26 18:07:20 UTC 2019
364 Tue Mar 26 18:07:30 UTC 2019
383 Tue Mar 26 18:07:40 UTC 2019
382 Tue Mar 26 18:07:50 UTC 2019
363 Tue Mar 26 18:08:00 UTC 2019
381 Tue Mar 26 18:08:10 UTC 2019
388 Tue Mar 26 18:08:20 UTC 2019
386 Tue Mar 26 18:08:30 UTC 2019
398 Tue Mar 26 18:08:40 UTC 2019
383 Tue Mar 26 18:08:50 UTC 2019
394 Tue Mar 26 18:09:00 UTC 2019
396 Tue Mar 26 18:09:10 UTC 2019
396 Tue Mar 26 18:09:20 UTC 2019
403 Tue Mar 26 18:09:30 UTC 2019
386 Tue Mar 26 18:09:40 UTC 2019
382 Tue Mar 26 18:09:50 UTC 2019
387 Tue Mar 26 18:10:00 UTC 2019
389 Tue Mar 26 18:10:10 UTC 2019
381 Tue Mar 26 18:10:20 UTC 2019
396 Tue Mar 26 18:10:30 UTC 2019
384 Tue Mar 26 18:10:40 UTC 2019
377 Tue Mar 26 18:10:50 UTC 2019
399 Tue Mar 26 18:11:00 UTC 2019
410 Tue Mar 26 18:11:10 UTC 2019
378 Tue Mar 26 18:11:20 UTC 2019
426 Tue Mar 26 18:11:30 UTC 2019
401 Tue Mar 26 18:11:40 UTC 2019
387 Tue Mar 26 18:11:50 UTC 2019
392 Tue Mar 26 18:12:00 UTC 2019
407 Tue Mar 26 18:12:10 UTC 2019
405 Tue Mar 26 18:12:20 UTC 2019
367 Tue Mar 26 18:12:30 UTC 2019
397 Tue Mar 26 18:12:40 UTC 2019
389 Tue Mar 26 18:12:50 UTC 2019
381 Tue Mar 26 18:13:00 UTC 2019
395 Tue Mar 26 18:13:10 UTC 2019
417 Tue Mar 26 18:13:20 UTC 2019
409 Tue Mar 26 18:13:30 UTC 2019
396 Tue Mar 26 18:13:40 UTC 2019
371 Tue Mar 26 18:13:50 UTC 2019
367 Tue Mar 26 18:14:00 UTC 2019
390 Tue Mar 26 18:14:10 UTC 2019
362 Tue Mar 26 18:14:20 UTC 2019
381 Tue Mar 26 18:14:30 UTC 2019
367 Tue Mar 26 18:14:40 UTC 2019
375 Tue Mar 26 18:14:50 UTC 2019
388 Tue Mar 26 18:15:00 UTC 2019
373 Tue Mar 26 18:15:10 UTC 2019
390 Tue Mar 26 18:15:20 UTC 2019
412 Tue Mar 26 18:15:30 UTC 2019
393 Tue Mar 26 18:15:40 UTC 2019
353 Tue Mar 26 18:15:50 UTC 2019
411 Tue Mar 26 18:16:00 UTC 2019
422 Tue Mar 26 18:16:10 UTC 2019
386 Tue Mar 26 18:16:20 UTC 2019
389 Tue Mar 26 18:16:30 UTC 2019
353 Tue Mar 26 18:16:40 UTC 2019
418 Tue Mar 26 18:16:50 UTC 2019
367 Tue Mar 26 18:17:00 UTC 2019
386 Tue Mar 26 18:17:10 UTC 2019
399 Tue Mar 26 18:17:20 UTC 2019
389 Tue Mar 26 18:17:30 UTC 2019
388 Tue Mar 26 18:17:40 UTC 2019
361 Tue Mar 26 18:17:50 UTC 2019
410 Tue Mar 26 18:18:00 UTC 2019
384 Tue Mar 26 18:18:10 UTC 2019
387 Tue Mar 26 18:18:20 UTC 2019
402 Tue Mar 26 18:18:30 UTC 2019
417 Tue Mar 26 18:18:40 UTC 2019
397 Tue Mar 26 18:18:50 UTC 2019
398 Tue Mar 26 18:19:00 UTC 2019
386 Tue Mar 26 18:19:10 UTC 2019
401 Tue Mar 26 18:19:20 UTC 2019
385 Tue Mar 26 18:19:30 UTC 2019
390 Tue Mar 26 18:19:40 UTC 2019
391 Tue Mar 26 18:19:50 UTC 2019
390 Tue Mar 26 18:20:00 UTC 2019
401 Tue Mar 26 18:20:10 UTC 2019
404 Tue Mar 26 18:20:20 UTC 2019
414 Tue Mar 26 18:20:30 UTC 2019
390 Tue Mar 26 18:20:40 UTC 2019
392 Tue Mar 26 18:20:50 UTC 2019
355 Tue Mar 26 18:21:00 UTC 2019
382 Tue Mar 26 18:21:10 UTC 2019
340 Tue Mar 26 18:21:20 UTC 2019
346 Tue Mar 26 18:21:30 UTC 2019
362 Tue Mar 26 18:21:40 UTC 2019
435 Tue Mar 26 18:21:50 UTC 2019
385 Tue Mar 26 18:22:00 UTC 2019
387 Tue Mar 26 18:22:10 UTC 2019
376 Tue Mar 26 18:22:20 UTC 2019
384 Tue Mar 26 18:22:30 UTC 2019
385 Tue Mar 26 18:22:40 UTC 2019
346 Tue Mar 26 18:22:50 UTC 2019
401 Tue Mar 26 18:23:00 UTC 2019
380 Tue Mar 26 18:23:10 UTC 2019
369 Tue Mar 26 18:23:20 UTC 2019
397 Tue Mar 26 18:23:30 UTC 2019
390 Tue Mar 26 18:23:40 UTC 2019
410 Tue Mar 26 18:23:50 UTC 2019
372 Tue Mar 26 18:24:00 UTC 2019
413 Tue Mar 26 18:24:10 UTC 2019
373 Tue Mar 26 18:24:20 UTC 2019
409 Tue Mar 26 18:24:30 UTC 2019
384 Tue Mar 26 18:24:40 UTC 2019
394 Tue Mar 26 18:24:50 UTC 2019
368 Tue Mar 26 18:25:00 UTC 2019
372 Tue Mar 26 18:25:10 UTC 2019
395 Tue Mar 26 18:25:20 UTC 2019
407 Tue Mar 26 18:25:30 UTC 2019
438 Tue Mar 26 18:25:40 UTC 2019
424 Tue Mar 26 18:25:50 UTC 2019
375 Tue Mar 26 18:26:00 UTC 2019
384 Tue Mar 26 18:26:10 UTC 2019
405 Tue Mar 26 18:26:20 UTC 2019
367 Tue Mar 26 18:26:30 UTC 2019
376 Tue Mar 26 18:26:40 UTC 2019
384 Tue Mar 26 18:26:50 UTC 2019
417 Tue Mar 26 18:27:00 UTC 2019
383 Tue Mar 26 18:27:10 UTC 2019
373 Tue Mar 26 18:27:20 UTC 2019
415 Tue Mar 26 18:27:30 UTC 2019
357 Tue Mar 26 18:27:40 UTC 2019
398 Tue Mar 26 18:27:50 UTC 2019
411 Tue Mar 26 18:28:00 UTC 2019
401 Tue Mar 26 18:28:10 UTC 2019
402 Tue Mar 26 18:28:20 UTC 2019
405 Tue Mar 26 18:28:30 UTC 2019
419 Tue Mar 26 18:28:40 UTC 2019
352 Tue Mar 26 18:28:50 UTC 2019
389 Tue Mar 26 18:29:00 UTC 2019
394 Tue Mar 26 18:29:10 UTC 2019
372 Tue Mar 26 18:29:20 UTC 2019
412 Tue Mar 26 18:29:30 UTC 2019
375 Tue Mar 26 18:29:40 UTC 2019
362 Tue Mar 26 18:29:50 UTC 2019
398 Tue Mar 26 18:30:00 UTC 2019
374 Tue Mar 26 18:30:10 UTC 2019
416 Tue Mar 26 18:30:20 UTC 2019
380 Tue Mar 26 18:30:30 UTC 2019
355 Tue Mar 26 18:30:40 UTC 2019
414 Tue Mar 26 18:30:50 UTC 2019
374 Tue Mar 26 18:31:00 UTC 2019
391 Tue Mar 26 18:31:10 UTC 2019
379 Tue Mar 26 18:31:20 UTC 2019
389 Tue Mar 26 18:31:30 UTC 2019
401 Tue Mar 26 18:31:40 UTC 2019
389 Tue Mar 26 18:31:50 UTC 2019
367 Tue Mar 26 18:32:00 UTC 2019
396 Tue Mar 26 18:32:10 UTC 2019
391 Tue Mar 26 18:32:20 UTC 2019
376 Tue Mar 26 18:32:30 UTC 2019
371 Tue Mar 26 18:32:40 UTC 2019
392 Tue Mar 26 18:32:50 UTC 2019
381 Tue Mar 26 18:33:00 UTC 2019
388 Tue Mar 26 18:33:10 UTC 2019
403 Tue Mar 26 18:33:20 UTC 2019
384 Tue Mar 26 18:33:30 UTC 2019
405 Tue Mar 26 18:33:40 UTC 2019
427 Tue Mar 26 18:33:50 UTC 2019
371 Tue Mar 26 18:34:00 UTC 2019
392 Tue Mar 26 18:34:10 UTC 2019
401 Tue Mar 26 18:34:20 UTC 2019
398 Tue Mar 26 18:34:30 UTC 2019
410 Tue Mar 26 18:34:40 UTC 2019
430 Tue Mar 26 18:34:50 UTC 2019
423 Tue Mar 26 18:35:00 UTC 2019
393 Tue Mar 26 18:35:10 UTC 2019
403 Tue Mar 26 18:35:20 UTC 2019
426 Tue Mar 26 18:35:30 UTC 2019
387 Tue Mar 26 18:35:40 UTC 2019
414 Tue Mar 26 18:35:50 UTC 2019
383 Tue Mar 26 18:36:00 UTC 2019
367 Tue Mar 26 18:36:10 UTC 2019
400 Tue Mar 26 18:36:20 UTC 2019
380 Tue Mar 26 18:36:30 UTC 2019
402 Tue Mar 26 18:36:40 UTC 2019
402 Tue Mar 26 18:36:50 UTC 2019
369 Tue Mar 26 18:37:00 UTC 2019
406 Tue Mar 26 18:37:10 UTC 2019
422 Tue Mar 26 18:37:20 UTC 2019
374 Tue Mar 26 18:37:30 UTC 2019
394 Tue Mar 26 18:37:40 UTC 2019
404 Tue Mar 26 18:37:50 UTC 2019
384 Tue Mar 26 18:38:00 UTC 2019
384 Tue Mar 26 18:38:10 UTC 2019
392 Tue Mar 26 18:38:20 UTC 2019
382 Tue Mar 26 18:38:30 UTC 2019
367 Tue Mar 26 18:38:40 UTC 2019
398 Tue Mar 26 18:38:50 UTC 2019
369 Tue Mar 26 18:39:00 UTC 2019
394 Tue Mar 26 18:39:10 UTC 2019
371 Tue Mar 26 18:39:20 UTC 2019
382 Tue Mar 26 18:39:30 UTC 2019
390 Tue Mar 26 18:39:40 UTC 2019
427 Tue Mar 26 18:39:50 UTC 2019
391 Tue Mar 26 18:40:00 UTC 2019
403 Tue Mar 26 18:40:10 UTC 2019
393 Tue Mar 26 18:40:20 UTC 2019
402 Tue Mar 26 18:40:30 UTC 2019
398 Tue Mar 26 18:40:40 UTC 2019
414 Tue Mar 26 18:40:50 UTC 2019
398 Tue Mar 26 18:41:00 UTC 2019
415 Tue Mar 26 18:41:10 UTC 2019
349 Tue Mar 26 18:41:20 UTC 2019
369 Tue Mar 26 18:41:30 UTC 2019
368 Tue Mar 26 18:41:40 UTC 2019
415 Tue Mar 26 18:41:50 UTC 2019
393 Tue Mar 26 18:42:00 UTC 2019
402 Tue Mar 26 18:42:10 UTC 2019
380 Tue Mar 26 18:42:20 UTC 2019
388 Tue Mar 26 18:42:30 UTC 2019
421 Tue Mar 26 18:42:40 UTC 2019
374 Tue Mar 26 18:42:50 UTC 2019
378 Tue Mar 26 18:43:00 UTC 2019
363 Tue Mar 26 18:43:10 UTC 2019
444 Tue Mar 26 18:43:20 UTC 2019
423 Tue Mar 26 18:43:30 UTC 2019
368 Tue Mar 26 18:43:40 UTC 2019
410 Tue Mar 26 18:43:50 UTC 2019
483 Tue Mar 26 18:44:00 UTC 2019
384 Tue Mar 26 18:44:10 UTC 2019
413 Tue Mar 26 18:44:20 UTC 2019
401 Tue Mar 26 18:44:30 UTC 2019
397 Tue Mar 26 18:44:40 UTC 2019
394 Tue Mar 26 18:44:50 UTC 2019
384 Tue Mar 26 18:45:00 UTC 2019
382 Tue Mar 26 18:45:10 UTC 2019
384 Tue Mar 26 18:45:20 UTC 2019
381 Tue Mar 26 18:45:30 UTC 2019
416 Tue Mar 26 18:45:40 UTC 2019
406 Tue Mar 26 18:45:50 UTC 2019
368 Tue Mar 26 18:46:00 UTC 2019
420 Tue Mar 26 18:46:10 UTC 2019
381 Tue Mar 26 18:46:20 UTC 2019
444 Tue Mar 26 18:46:30 UTC 2019
383 Tue Mar 26 18:46:40 UTC 2019
383 Tue Mar 26 18:46:50 UTC 2019
357 Tue Mar 26 18:47:00 UTC 2019
376 Tue Mar 26 18:47:10 UTC 2019
374 Tue Mar 26 18:47:20 UTC 2019
411 Tue Mar 26 18:47:30 UTC 2019
421 Tue Mar 26 18:47:40 UTC 2019
386 Tue Mar 26 18:47:50 UTC 2019
395 Tue Mar 26 18:48:00 UTC 2019
415 Tue Mar 26 18:48:10 UTC 2019
389 Tue Mar 26 18:48:20 UTC 2019
374 Tue Mar 26 18:48:30 UTC 2019
392 Tue Mar 26 18:48:40 UTC 2019
370 Tue Mar 26 18:48:50 UTC 2019
365 Tue Mar 26 18:49:00 UTC 2019
410 Tue Mar 26 18:49:10 UTC 2019
396 Tue Mar 26 18:49:20 UTC 2019
406 Tue Mar 26 18:49:30 UTC 2019
406 Tue Mar 26 18:49:40 UTC 2019
353 Tue Mar 26 18:49:50 UTC 2019
426 Tue Mar 26 18:50:00 UTC 2019
403 Tue Mar 26 18:50:10 UTC 2019
389 Tue Mar 26 18:50:20 UTC 2019
378 Tue Mar 26 18:50:30 UTC 2019
383 Tue Mar 26 18:50:40 UTC 2019
433 Tue Mar 26 18:50:50 UTC 2019
359 Tue Mar 26 18:51:00 UTC 2019
414 Tue Mar 26 18:51:10 UTC 2019
382 Tue Mar 26 18:51:20 UTC 2019
383 Tue Mar 26 18:51:30 UTC 2019
386 Tue Mar 26 18:51:40 UTC 2019
406 Tue Mar 26 18:51:50 UTC 2019
361 Tue Mar 26 18:52:00 UTC 2019
375 Tue Mar 26 18:52:10 UTC 2019
419 Tue Mar 26 18:52:20 UTC 2019

per10s.png (784×2 px, 152 KB)

per100s.png (792×2 px, 70 KB)

Request rate reduced by 7-8 %

The global request rate to load.php dropped by 7-8 %, e.g. the mid-day values around 11 AM UTC are down from 25K/s (Tue last week) to 22K/s (today), and the daily peek of 33K/s is now down to 30K/s .

export-rate.png (1×2 px, 112 KB)

Looking at the breakdown by HTTP status, we can see this gain of fewer requests correlates with a reduction in requests that got a "304 Not Modified" response, without an increase in "200 OK" responses. In other words, these appear to be requests that previously were always unmodified, but the client confirmed them regardless. Whereas now the client is already aware of that without needing to ask the server.

export-status.png (1×2 px, 88 KB)

This impact is expected. Previously, we had a fixed 5-min looping window that would expire for everyone at the same time. This meant that no matter when your Wikipedia browsing session started, the cache would expire at the global 5-min interval, which could be as little as 1 minute or 10 seconds from now. After this change, it will always expire 5 minutes after your own personal window started, which means you'll make much fewer requests on average. And (more importantly) you'll be able to leverage that cache consistently for several page views within the first 5 minutes of the session.

In addition to affecting the first 5-min of any browsing session, it also means the next 5 minute window will start based on your personal browsing behaviour. That is, if you navigate from page C to page D and the cache expired in-between, it previously started counting at the expiry point (in response to another user elsewhere in the world). Now, it will start counting from your own navigation. Again, this means fewer requests overall and much better use of client-side caching.

Krinkle renamed this task from Expires header for load.php should be relative to request time instead of cache time to The 5min expires for load.php/startup should be relative to request time instead of cache time.Sep 19 2019, 8:57 PM
Krinkle renamed this task from The 5min expires for load.php/startup should be relative to request time instead of cache time to The 5min expiry for load.php/startup should be relative to request time instead of cache time.