Page MenuHomePhabricator

Create MediaViewer image varnish hit/miss ratio dashboard
Closed, ResolvedPublic

Details

Related Gerrit Patches:
analytics/multimedia/config : masterCalculate image cache miss ratio
analytics/multimedia : masterCalculate image cache miss ratio

Event Timeline

Tgr created this task.Dec 10 2014, 11:02 PM
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr added projects: MediaViewer, Multimedia, Analytics.
Tgr changed Security from none to None.
Tgr added a subscriber: Tgr.

This would complement T76035.

Tgr triaged this task as Medium priority.Dec 10 2014, 11:56 PM

Change 179771 had a related patch set uploaded (by Gergő Tisza):
Calculate image cache miss ratio

https://gerrit.wikimedia.org/r/179771

Patch-For-Review

Change 179778 had a related patch set uploaded (by Gergő Tisza):
Calculate image cache miss ratio

https://gerrit.wikimedia.org/r/179778

Patch-For-Review

Change 179771 merged by jenkins-bot:
Calculate image cache miss ratio

https://gerrit.wikimedia.org/r/179771

Gilles assigned this task to Tgr.Dec 16 2014, 9:55 AM
Gilles moved this task from Untriaged to Needs code review on the Multimedia board.
Gilles added a subscriber: Gilles.

Note that pre-rendered thumbnails will appear as a varnish miss. The first time they're requested they're in swift, but not in varnish.

Tgr added a comment.EditedDec 16 2014, 10:06 PM

Yeah, I didn't think of that. The Last-Modified header of thumbnails seems match when they were generated (Swift also adds an X-Timestamp header which seems to be the same). Maybe we should add that to our performance logging and assume a scaler miss if it is older than the time of sending the request? (Clock skew errors, yay.)

Tgr added a comment.Dec 16 2014, 10:08 PM

Or when the last-modified header is older than the date header minus the difference between local times for request and response? That's reasonably robust and we are collecting those times already.

In T78205#851804, @Tgr wrote:

Yeah, I didn't think of that. The Last-Modified header of thumbnails seems match when they were generated

Good catch! It will be very helpful to know the performance of thumbnails that were pregenerated but not copied from swift to varnish yet at the time of the request.

In T78205#851811, @Tgr wrote:

Or when the last-modified header is older than the date header minus the difference between local times for request and response? That's reasonably robust and we are collecting those times already.

That seems better, you can only ever use local time for relative time measurement. Some people have their clocks off by years.

Actually I see that there's a way to tell this only with headers, no need to calculate the local time difference. The "Age" header is the missing part of the puzzle. If the thumbnail is generated on the spot: Date - Last-Modified <= Age + 1 (the extra second is there because of rounding). If the thumbnail has been generated some time ago and just pulled from swift, Date - Last-Modified > Age + 1

The theory doesn't seem to hold true, the vast majority of varnish misses with a very small "Age" value have a very old Last-Modified value, regardless of when those files were uploaded. I think the explanations is that those thumbnails expire in varnish by not being accessed very often, then they're pulled from swift again when they're requested. Therefore old thumbnails can also be in that situation of being pulled from swift on the spot instead of being generated.

So while we can differentiate "true" misses (thumbnails have to be generated on the spot) from swift pulls thanks to Last-Modified, we can't tell if the swift pulls are happening in a prerendering scenario or a varnish expiry situation.

What's interesting in those findings, though, is that 99.34% of varnish misses are swift pulls, regardless of upload time. Which would suggest that unless we increase the duration thumbnails are retained for in Varnish, there isn't much of a performance gain to be had for misses. The thumbnails have almost all been generated a while ago and are at least in Swift.

It also means that prerendering only helps eliminate 0.66% of the varnish misses having to generate the thumbnail, although with recent files that ratio would probably be higher.

Also, while not that many varnish misses generating a thumbnail generation have been caught yet, they've all happened for files where the file upload time is older than the prerendering deployment: P173 Which is a confirmation that we're looking at the right information.

Gilles moved this task from Prototyping to Untriaged on the Multimedia board.Apr 6 2015, 9:22 AM
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:24 PM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptSep 4 2015, 6:24 PM

Mass-removing the Multimedia tag from MediaViewer tasks, as this is now being worked on by the Reading department, not Editing's Multimedia team.

Jdlrobson changed the task status from Open to Stalled.Sep 24 2015, 12:32 AM
Jdlrobson added a subscriber: Jdlrobson.

What's left to do here?
I'm a little confused. No activity since April. Please update and un-stall it :)

Tgr added a comment.Sep 24 2015, 1:49 AM

Done in https://grafana.wikimedia.org/#/dashboard/db/media I think? Although that's (Swift + Varnish) hit/miss, not pure Varnish.

Jdlrobson closed this task as Resolved.Jul 7 2016, 4:48 PM

Due to being vague and probably fixed.

Restricted Application added a project: Multimedia. · View Herald TranscriptMar 6 2018, 11:52 PM