@fgiunchedi would it be ok for production to pin that particular package to testing/unstable, or do we need to have it ported over to stretch-backports?
It's on Buster now: https://packages.debian.org/source/derpconf
It turns out to be quite common for load.php calls to take more than a minute: https://logstash.wikimedia.org/goto/788ca720a38ccbed8dab29adab7ac2ca
Fri, Jan 12
"The Web, the Users, and the MOS: Influence of HTTP/2 on User Experience" Enrico Bocchi, Luca De Cicco, Marco Mellia, Dario Rossi, 2017
"The duration perception of loading applications in smartphone: Effects of different loading types" Wenguo Zhao, Yan Ge, Weina Qu, Kan Zhang, Xianghong Sun 2017
Thu, Jan 11
This might be of interest: https://github.com/alvarcarto/url-to-pdf-api looks very similar, if not identical, to what you want to build. Could serve as inspiration at least.
Files that can be expensive to thumbnail are nothing new, giant images and multipage documents also fall in that category. It's not uncommon in production for giant TIFFs to take 20 seconds to render.
Wed, Jan 10
Tue, Jan 9
This is now blocked on the Mediawiki core change to add the ability to proxy requests to Thumbor from thumb.php
Doesn't seem like it fixed the issue with the percentage character in production. Oh well, back to the drawing board...
I'm going to leave the older papers aside for now and look for more relevant recent studies instead to complete the list. Dated technology, particularly for mobile, makes a lot of these older studies difficult to translate to current usage patterns.
2005: “Interaction in 4-Second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI” Antti Oulasvirta, Sakari Tamminen, Virpi Roto, and Jaana Kuorelahti, Interruptions in Human Computer Interaction
Kaaresoja T., Hoggan E., Anttila E. (2011) Playing with Tactile Feedback Latency in Touchscreen Interaction: Two Approaches. In: Campos P., Graham N., Jorge J., Nunes N., Palanque P., Winckler M. (eds) Human-Computer Interaction – INTERACT 2011. INTERACT 2011.
Mon, Jan 8
As for thumbnails, it's not that it's hard to implement, but you'll create a lot of extra purge traffic and cache invalidation for a threat that is currently only theoretical. Consult the Traffic team about this, imho it's not a worthy tradeoff until the loophole is really exploited.
Thumbor isn't involved with transcodes, only thumbnails. Taking care of transcode logic for this should all happen within Mediawiki, afaik.
Change merged. Not sure this is worth bumping the upstream Debian package for.
Found one that 500s a the moment:
Actually, from your last comment, the 1st and 3rd one come out as 400s for me right now, and the 2nd one works. It's possible that the ones with % in them shouldn't be 400s, but I wonder why they showed up in the Varnish logs as 500s.
Fixed on the Stretch branch.
Given that the vast majority of the abuse was with video files, transcode support seems like a must have. As it stands, the workaround is very easy for any original that gets blocked with this new feature: just watch a transcode listed on the file page.
Sat, Jan 6
Thu, Jan 4
The percent character without being an encoded character code seems like the likely culprit for most of them.
Wed, Jan 3
Most affected files have been fixed. The occurrences of that error dropped like a rock in logstash. There are a couple of hits still, but probably for other reasons (url encoding?):
Here's my review of "A Study on Tolerable Waiting Time: How long Are Web Users Willing to Wait?” (PDF), Fiona Fui-Hoon Nah, Behaviour and Information Technology, 2004, cited in the RAIL article.
Here's my review of "Response Times: The 3 Important Limits" by Jakob Nielsen, 1993, cited in the RAIL article.
This currently fixes some of the files showing up in logstash, but not all.
Tue, Jan 2
Testing this on thumbor1001, seeking for the midpoint with ffmpeg fails silently. The status code is 0, but no screenshot is generated. If the fallback of seeking to 0 kicked in, it would work.
Cape_Town_under_the_clouds.webm is playable, but for me it skips straight to 11:12 when I open it in Chrome. Which probably suggests something unusual about the file or its keyframes. On VLC it plays for 33 seconds, instead of the announced 11 minutes duration. Which is consistent with the approximate amount of time it plays in Chrome. This file looks corrupt.
Here's my review of Response Time in Man-Computer Conversational Transactions" by Robert B; Miller, Fall Joint Computer Conference 1968, the first paper cited in the RAIL article.
Looking at the new breakdown dashboard: https://grafana.wikimedia.org/dashboard/db/backend-save-timing-breakdown?refresh=5m&orgId=1
I think it could be useful as-is, with the understanding that it only works well on a specific set of browsers, because it extracts a potentially useful metric from a large set of very fine-grained metrics we don't collect currently and probably never will (precisely because it's too fine-grained).
Sat, Dec 23
Thu, Dec 21
Out of curiosity, do all roles provision correctly on master? (Jessie)