We tend to point to guidelines like RAIL in performance discussions, but the figures put forward in these guidelines rarely, if ever, reference the underlying research. There might not be any data or study that actually prove the existence of the thresholds described or their values.
We should formally review what's been done to make up our own opinion about performance speed guidelines, or even come up with guidelines of our own, or create new studies that would fill the gaps.
It would be interesting to specifically look for studies based on real data/research that answers the following questions:
- What's the speed threshold for what feels instantaneous? In what context?
- What's the speed threshold for something so slow that it becomes frustrating to the user? In what context?
- In what contexts has "faster is better" been demonstrated? Is there any other context where slower has been proven to be better?
- What's the distribution of performance perception between different people?
We should be particularly wary of studies that take mental speed/time thresholds for granted. I've seen it to be quite common that previous research papers are referenced on that matter, but the original papers didn't base those thresholds on any research, they were actually part of their unconfirmed hypothesis. Some of the more recent papers are quite critical of that, observing that those "magic numbers" keep getting propagated but have not necessarily been proven.
Below is a working list of research papers of interest, that keeps expanding as we explore them and their citations. The starting point was the list quoted in the RAIL announcement, as well as searches for perceived performance keywords in google scholar.
Miller, R.B., 1968, Response time in man-computer conversational transaction. Proceedings of AFIPS Fall Joint Computer Conference, 33, 267-277. Guynes, J.L., 1988, Impact of system response time on state anxiety. Communications of the ACM, 31(3), 342-347. Nielsen, J., 1993, Response times: the three important limits. Excerpt from Chapter 5 of Usability Engineering by Jakob Nielsen, Academic Press, 1993. Jacko, J.A., Sears, A. and Borella, M.S., 2000, The effect of network delay and media on user perceptions of web resources. Behaviour and Information Technology, 19(6), 427-439. Galletta, D., Henry, R., McCoy, S. and Polak, P., 2002, Web site delays: how slow can you go?” Presented at the First Annual Pre-ICIS Workshop on HCI Research in MIS, 14 December 2002 forthcoming in Journal of Association for Information Systems. Selvidge, P., 2003, Examining tolerance for online delays. Usability News, 5(1). Rose, G.M., Evaristo, R. and Straub, D., 2003, Culture and consumer responses to web download time: a four-continent study of mono and polychronism. IEEE Transactions on Engineering Management, 50(1), 31-44. Nah, Fiona, "A Study on Tolerable Waiting Time: How Long Are Web Users Willing to Wait?" (2003). AMCIS 2003 Proceedings. 285. 2005: “Interaction in 4-Second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI” Antti Oulasvirta, Sakari Tamminen, Virpi Roto, and Jaana Kuorelahti, Interruptions in Human Computer Interaction 2012: “Characterizing Web Use on Smartphones”, Chad C. Tossell, Philip Kortum, Ahmad Rahmati, Clayton Shepard, Lin Zhong, Conference on Human Factors in Computing Systems 2012 Kaaresoja T., Hoggan E., Anttila E. (2011) Playing with Tactile Feedback Latency in Touchscreen Interaction: Two Approaches. In: Campos P., Graham N., Jorge J., Nunes N., Palanque P., Winckler M. (eds) Human-Computer Interaction – INTERACT 2011. INTERACT 2011. "Quantify Perceived Performance" Ingmar Verheij 2011 "A methodology for the evaluation of high response time on E-commerce users and sales" Nicolas Poggi, David Carrera, Ricard Gavaldà, Eduard Ayguadé, Jordi Torres 2012 "Improving the Human–Computer Dialogue With Increased Temporal Predictability" Florian Weber, Carola Haering, Roland Thomaschke 2013 "Demystifying Page Load Performance with WProf" Wang, Balasubramanian, Krishnamurthy, Wetherall 2013 "Stuck in traffic: how temporal delays affect search behaviour" David Maxwell, Leif Azzopardi 2014 "The Impact of Waiting Time Distributions on QoE of Task-Based Web Browsing Sessions" Islam Nazrul, Elepe Vijaya John David 2014 "Perception of Delay and Attitude toward Feedback Display: An Exploration into Downloaders’ Demographics" Chatpong Tangmanee, Pawarat Nontasil 2014 "Towards Better Measurement of Attention and Satisfaction in Mobile Search" Lagun, Hsieh, Webster, Navalpakkam 2014 "Defining Standards for Web Page Performance in Business Applications" Garret Rempel 2015 "User-Acceptance of Latency in Touch Interactions" Walter Ritter, Guido Kempter, Tobias Werner 2015 "Klotski: Reprioritizing Web Content to Improve User Experience on Mobile Devices" Butkiewicz, Wang, Wu, Madhyastha, Sekar 2015 "Measuring the Quality of Experience of Web users" Bocchi, De Cicco, Rossi 2016 "What slows you down? Your network or your device?" Steiner, Gao 2016 "Critical CSS Rules — Decreasing time to first render by inlining CSS rules for over-the-fold elements" Jovanovski, Zaytsev 2016 "Oak: User-Targeted Web Performance" Flores, Wenzel, Kuzmanovic 2016 "EYEORG: A Platform For Crowdsourcing Web Quality Of Experience Measurements" Varvello, Blackburn, Naylor, Papagiannaki, 2016 "Search Result Prefetching on Desktop and Mobile" Ryen W. White, Fernando Diaz, Qi Guo, 2017 "The duration perception of loading applications in smartphone: Effects of different loading types" Wenguo Zhao, Yan Ge, Weina Qu, Kan Zhang, Xianghong Sun 2017 "The Web, the Users, and the MOS: Influence of HTTP/2 on User Experience" Enrico Bocchi, Luca De Cicco, Marco Mellia, Dario Rossi, 2017 "Are 100 ms Fast Enough? Characterizing Latency Perception Thresholds in Mouse-Based Interaction" Valentin Forch, Thomas Franke, Nadine Rauh, Josef F. Krems 2017 "Perceived Performance of Top Retail Webpages In the Wild: Insights from Large-scale Crowdsourcing of Above-the-Fold QoE" Gao, Dey, Ahammad 2017 "PAIN: A Passive Web Speed Indicator for ISPs" Marco Mellia, Idilio Drago, Martino Trevisan 2017 "Reordering Webpage Objects for Optimizing Quality-of-Experience" Li, Zhao, Min, Duan, Ni, Zhao 2017 "Improving User Perceived Page Load Times Using Gaze" Kelton, Ryoo, Balasubramanian, Das 2017 "Vesper: Measuring Time-to-Interactivity for Modern Web Pages" Netravali, Nathan, Mickens, Balakrishnan 2017 "Narrowing the gap between QoS metrics and Web QoE using Above-the-fold metrics" Diego Neves da Hora, Alemnew Sheferaw Asrese, Vassilis Christophides, Renata Teixeira, Dario Rossi 2018
It would be nice to put together a literature review at the end of this, summarizing the different papers read and drawing connections between them.