Use the same technique as EventLogging. Instead of going to a backend like EL, these requests go nowhere and are simply logged as part of the Varnish logs.
See the relevant code used for EL: https://github.com/wikimedia/operations-puppet/blob/production/templates/varnish/bits.inc.vcl.erb#L24
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Milimetric | T77541 Per-file view stats | |||
| Resolved | • Gilles | T89088 Set up varnish 204 beacon endpoint for virtual media views and use it in Media Viewer |
Change 190821 had a related patch set uploaded (by Gilles):
Set up beacon endpoint for virtual media views
Change 190823 had a related patch set uploaded (by Gilles):
Record virtual image views
I'm not sure why this is necessary -- couldn't you just count the image requests themselves? As far as I know, we have the full request logs in Hadoop, so this should be possible.
Images are preloaded, which means that an image can be requested frequently just because it is next to a popular image. Request counts are also, to a smaller extent, distorted by caching (which is itself somewhat random as cache TTL heuristics are probably based on how long ago the image was uploaded).
While I can see how we might be backed into this corner to get data we need, it seems wrong on so many levels to create additional requests in order to track the requests we're already getting.
The use of virtual pageviews is becoming the norm in the industry, because there's no way around it for modern apps. When you start making things service-oriented, your web interface being a thin layer using pushstate and your backend serving multiple very different frontends and you optimize your performance to leverage the browser or device's smart prefetching, come up with a preloading strategy of your own, bundle unrelated requests together when they happen close in time, etc. there's no other way around recording pageviews independently of backend requests.
The fact that we have a mostly outdated ecosystem that for the most part still relies on full, oldschool, html pageviews with no end-user performance concerns addressed, shouldn't be the reason why these needs are pushed back. The more heavily front-end-oriented applications get built, the more you should expect others to need virtual pageviews.
Also, had EventLogging supported the load for unsampled recording of those events, we would have just used that and nobody would have cared. It's just that EL seems better suited for sampled measurements, which have a completely different purpose, and we shouldn't try to shove everything under EL, I think.
Don't let the fact that some backend requests seem to be related fool you. There's no way to tell when someone has viewed something from their local cache other than by recording a virtual view. The next and previous buttons in Media Viewer are used a lot, it's the whole point of the feature, and whenever you do go to the next or previous image, there's a high likelihood that you'll be looking at a preloaded image, already in your browser's cache. And there's a high percentage of preloaded images that you'll never look at. There's no way to generate remotely accurate per-file stats in Media Viewer by looking at the image requests themselves.
We'll also be tracking for the first time how long people keep the image in view for, for free (just an extra parameter passed to the beacon url). Which is something that can't be inferred from the existing backend requests and therefore couldn't be done without an extra request. We could be doing more, that's just an example of what's possible once you decouple the backend requests and recording the views.
Change 190821 abandoned by Ori.livneh:
Set up beacon endpoint for virtual media views
Reason:
obsoleted by https://gerrit.wikimedia.org/r/#/c/192370/
Change 196180 had a related patch set uploaded (by Gilles):
Make Media Viewer record virtual image views