Change Details

We tend to point to guidelines like RAIL in performance discussions, but the figures put forward in these guidelines rarely, if ever, reference the underlying research. There might not be any data or study that actually prove the existence of the thresholds described or their values. We should formally review what's been done to make up our own opinion about performance speed guidelines, or even come up with guidelines of our own, or create new studies that would fill the gaps. It would be interesting to specifically look for studies based on real data/research that answers the following questions: - What's the speed threshold for what feels instantaneous? In what context? - What's the speed threshold for something so slow that it becomes frustrating to the user? In what context? - In what contexts has "faster is better" been demonstrated? Is there any other context where slower has been proven to be better? - What's the distribution of performance perception between different people? We should be particularly wary of studies that take mental speed/time thresholds for granted. I've seen it to be quite common that previous research papers are referenced on that matter, but the original papers didn't base those thresholds on any research, they were actually part of their unconfirmed hypothesis. Some of the more recent papers are quite critical of that, observing that those "magic numbers" keep getting propagated but have not necessarily been proven. Below is a working list of research papers of interest, that keeps expanding as we explore them and their citations. The starting point was [[ https://www.smashingmagazine.com/2015/10/rail-user-centric-model-performance/ | the list quoted in the RAIL announcement ]]. - ~~[[ http://theixdlibrary.com/pdf/Miller1968.pdf | Miller, R.B., 1968, Response time in man-computer conversational transaction. Proceedings of AFIPS Fall Joint Computer Conference, 33, 267-277. ]]~~ - Rule, S.J., Curtis, D.W. and Markley, R.P., 1970, Input and output transformations from magnitude estimation. Journal of Experimental Psychology, 86(1), 343-349. - Hornik, J., 1986, Subjective vs. objective time measures: a note on the perception of time in consumer behavior. Journal of Consumer Research, 11, 615-618. - Guynes, J.L., 1988, Impact of system response time on state anxiety. Communications of the ACM, 31(3), 342-347. - [[ https://www.researchgate.net/profile/Richard_Larson5/publication/304582002_Prescription_for_the_Waiting_in_Line_Blues_Entertain_Enlighten_Engage/links/5774627708aeb9427e2422c4/Prescription-for-the-Waiting-in-Line-Blues-Entertain-Enlighten-Engage.pdf | Katz, K., Larson, B. and Larson, R., 1991, Prescription for the waiting in line blues: entertain, enlighten and engage. Sloan Management Review, 32(2), 44-53. ]] - ~~[[ http://www.useit.com/papers/responsetime.html | Nielsen, J., 1993, Response times: the three important limits. Excerpt from Chapter 5 of Usability Engineering by Jakob Nielsen, Academic Press, 1993. ]]~~ - Taylor, S., 1994, Waiting for service: the relationship between delays and evaluations of service. Journal of Marketing, 58(2), 56-69. - Hui, M.K. and Tse, D., 1996, What to tell consumers in waits of different lengths: an integrative model of service evaluation. Journal of Marketing, 60(2), 81-90. - Hui, M.K. and Zhou, L., 1996, How does waiting duration information influence customers’ reactions to waiting for services? Journal of Applied Social Psychology, 26(19), 1702-1717. - Ramsay, J., Barbesi, A. and Preece, J., 1998, A psychological investigation of long retrieval times on the world wide web. Interacting with Computers, 10(1), 77-86. - Dellaert, B.G.C. and Kahn, B.E., 1999, How tolerable is delay? Consumers’ evaluations of Internet Web sites after waiting. Journal of Interactive Marketing, 13(1), 41-54. - [[ http://usabilitynews.org/how-long-is-too-long-to-wait-for-a-website-to-load/ | Selvidge, P., 1999, How long is too long for a website to load? Usability News, 1(2). ]] - Jacko, J.A., Sears, A. and Borella, M.S., 2000, The effect of network delay and media on user perceptions of web resources. Behaviour and Information Technology, 19(6), 427-439. - [[ http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.99.2770 | Hoxmeier, J.A. and DiCesare, C., 2000, System response time and user satisfaction: an experimental study of browser-based applications. Proceedings of the Americas Conference on Information Systems, 10-13 August 2000 (Long Beach, California: Association for Information Systems), pp. 140-145. ]] - Weinberg, B.D., 2000, Don’t keep your Internet customers waiting too long at the (virtual) front door. Journal of Interactive Marketing, 14(10), 30-38. - [[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.907.7268&rep=rep1&type=pdf | Rose, G.M. and Straub, D.W., 2001, The effect of download time on consumer attitude toward the e-service retailer. e-Service Journal, 1, 55-76. ]] - [[ http://www.mediajournal.org/index.php/jmm/article/view/167/50 | Rose, G.M., Lees, J. and Meuter, M.L., 2001, A refined view of download time impacts on e-consumer attitudes and patronage intentions toward e-retailers. International Journal of Media Management, 3, 105-111. ]] - ~~Zona Research, 2001, The need for speed II. Zona Market Bulletin, 5, April 2001.~~ (can't find it) - Antonides, G., Verhoef, P.C., van Aalst, M., 2002, Consumer perception and evaluation of waiting time: a field experiment. Journal of Consumer Psychology, 12(3), 193-202. - [[ http://melody.syr.edu/hci/pre_icis02_wksp/pres/11_Galletta.ppt | Galletta, D., Henry, R., McCoy, S. and Polak, P., 2002, Web site delays: how slow can you go?” Presented at the First Annual Pre-ICIS Workshop on HCI Research in MIS, 14 December 2002 forthcoming in Journal of Association for Information Systems. ]] - ~~[[ http://usabilitynews.org/examining-tolerance-for-online-delays/ | Selvidge, P., 2003, Examining tolerance for online delays. Usability News, 5(1). ]]~~ - Rose, G.M., Evaristo, R. and Straub, D., 2003, Culture and consumer responses to web download time: a four-continent study of mono and polychronism. IEEE Transactions on Engineering Management, 50(1), 31-44. - ~~[[ http://sighci.org/uploads/published_papers/bit04/BIT_Nah.pdf | Nah, Fiona, "A Study on Tolerable Waiting Time: How Long Are Web Users Willing to Wait?" (2003). AMCIS 2003 Proceedings. 285. ]]~~ - ~~[[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.155&rep=rep1&type=pdf | 2005: “Interaction in 4-Second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI” Antti Oulasvirta, Sakari Tamminen, Virpi Roto, and Jaana Kuorelahti, Interruptions in Human Computer Interaction ]]~~ - [[ http://ianspiro.com/chi2008.pdf | "Eye-mouse coordination patterns on web search results pages" Rodden, Fu, Aula, Spiro 2008 ]] - "The influence of cognitive load, physical load and system response time on user productivity and satisfaction with computer systems" Dabrowski, James R. 2010 - "WebProphet: automating performance prediction for web services" Li, Zhang, Zhu, Chen, Greenberg, Wang 2010 - [[ http://isr.cmu.edu/doc/tolia06-ieee.pdf | 2006: “Quantifying Interactive User Experience on Thin Clients”, Niraj Tolia, David G. Andersen, and M. Satyanarayanan, The Internet Suspend/Resume Project, Carnegie Mellon ]] - ~~[[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.366.1170&rep=rep1&type=pdf | 2012: “Characterizing Web Use on Smartphones”, Chad C. Tossell, Philip Kortum, Ahmad Rahmati, Clayton Shepard, Lin Zhong, Conference on Human Factors in Computing Systems 2012 ]]~~ - ~~[[ https://link.springer.com/chapter/10.1007%2F978-3-642-23771-3_42 | Kaaresoja T., Hoggan E., Anttila E. (2011) Playing with Tactile Feedback Latency in Touchscreen Interaction: Two Approaches. In: Campos P., Graham N., Jorge J., Nunes N., Palanque P., Winckler M. (eds) Human-Computer Interaction – INTERACT 2011. INTERACT 2011. ]]~~ - ~~[[ http://www.ingmarverheij.com/wp-content/uploads/downloads/2011/09/Whitepaper-Quantifying-Perceived-Performance-v1.0.pdf | "Quantify Perceived Performance" Ingmar Verheij 2011 ]]~~ - ~~[[ https://pdfs.semanticscholar.org/5c5c/7a2b067cddea7dedf3f66dce8b3f93558b72.pdf | "Improving the Human–Computer Dialogue With Increased Temporal Predictability" Florian Weber, Carola Haering, Roland Thomaschke 2013 ]]~~ - ~~[[ https://homes.cs.washington.edu/~arvind/papers/wprof.pdf | "Demystifying Page Load Performance with WProf" Wang, Balasubramanian, Krishnamurthy, Wetherall 2013 ]]~~ - "A methodology for the evaluation of high response time on E-commerce users and sales" Nicolas Poggi, David Carrera, Ricard Gavaldà, Eduard Ayguadé, Jordi Torres 2014 - ~~[[ http://www.dmax.org.uk/uploads/publications/iiix2014cost_performance_interaction.pdf | "Stuck in traffic: how temporal delays affect search behaviour" David Maxwell, Leif Azzopardi 2014 ]]~~ - ~~[[ https://www.diva-portal.org/smash/get/diva2:832423/FULLTEXT01.pdf | "The Impact of Waiting Time Distributions on QoE of Task-Based Web Browsing Sessions" Islam Nazrul, Elepe Vijaya John David 2014 ]]~~ - "Perception of Delay and Attitude toward Feedback Display: An Exploration into Downloaders’ Demographics" Chatpong Tangmanee, Pawarat Nontasil 2014 - [[ https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chow.pdf | "The Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services" Chow, Meisner, Flinn, Peek, Wenisch 2014 ]] - ~~[[ https://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/43224.pdf | "Towards Better Measurement of Attention and Satisfaction in Mobile Search" Lagun, Hsieh, Webster, Navalpakkam 2014]]~~ - ~~[[ https://research.spec.org/icpe_proceedings/2015/icpe/p245.pdf | "Defining Standards for Web Page Performance in Business Applications" Garret Rempel 2015 ]]~~ - ~~"User-Acceptance of Latency in Touch Interactions" Walter Ritter, Guido Kempter, Tobias Werner 2015~~ - ~~[[ https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-butkiewicz.pdf | "Klotski: Reprioritizing Web Content to Improve User Experience on Mobile Devices" Butkiewicz, Wang, Wu, Madhyastha, Sekar 2015 ]]~~ - [[ https://dspace.mit.edu/bitstream/handle/1721.1/100162/928920895-MIT.pdf?sequence=1 | "DepTracker: Tracking JavaScript Dependencies on the Web" Goyal 2015 ]] - "Predicting web page performance level based on web page characteristics" Zhou, Zhang, Zhou, Li 2015 - ~~[[ https://perso.telecom-paristech.fr/~drossi/paper/rossi16internet-qoe.pdf | "Measuring the Quality of Experience of Web users" Bocchi, De Cicco, Rossi 2016 ]]~~ - [[ https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-netravali.pdf | "Polaris: Faster Page Loads Using Fine-grained Dependency Tracking" Netravali, Goyal, Mickens, Balakrishnan 2016 ]] - [[ https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-wang-xiao-sophia.pdf | "Speeding up Web Page Loads with Shandian" Wang, Krishnamurthy, Wetherall 2016 ]] - [[ https://arxiv.org/pdf/1602.03273.pdf | "YTrace: End-to-end Performance Diagnosis in Large Cloud and Content Providers" Kanuparthy, Dai, Pathak, Samal, Benson, Ghasemi, Narayan 2016 ]] - ~~[[ https://arxiv.org/pdf/1603.02293.pdf | "What slows you down? Your network or your device?" Steiner, Gao 2016 ]]~~ - ~~[[ http://sattose.wdfiles.com/local--files/2016:alltalks/SATTOSE2016_paper_13.pdf | "Critical CSS Rules — Decreasing time to first render by inlining CSS rules for over-the-fold elements" Jovanovski, Zaytsev 2016 ]]~~ - ~~[[ https://www.mccormick.northwestern.edu/eecs/documents/tech-reports/2016/2016-10-oak-user-targeted-web-performance.pdf | "Oak: User-Targeted Web Performance" Flores, Wenzel, Kuzmanovic 2016 ]]~~ - ~~[[ https://davidtnaylor.com/eyeorg.pdf | "EYEORG: A Platform For Crowdsourcing Web Quality Of Experience Measurements" Varvello, Blackburn, Naylor, Papagiannaki, 2016 ]]~~ - [[ https://www.usenix.org/system/files/conference/atc16/atc16-paper-vesuna.pdf | "Caching Doesn’t Improve Mobile Web Performance (Much)" Vesuna, Scott, Buettner, Piatek, Krishnamurthy, Shenker 2016 ]] - ~~[[ http://fernando.diaz.nyc/WhiteTOIS2016.pdf | "Search Result Prefetching on Desktop and Mobile" Ryen W. White, Fernando Diaz, Qi Guo, 2017 ]]~~ - ~~"The duration perception of loading applications in smartphone: Effects of different loading types" Wenguo Zhao, Yan Ge, Weina Qu, Kan Zhang, Xianghong Sun 2017~~ - ~~[[ https://perso.telecom-paristech.fr/drossi/paper/rossi17pam.pdf | "The Web, the Users, and the MOS: Influence of HTTP/2 on User Experience" Enrico Bocchi, Luca De Cicco, Marco Mellia, Dario Rossi, 2017 ]]~~ - "Are 100 ms Fast Enough? Characterizing Latency Perception Thresholds in Mouse-Based Interaction" Valentin Forch, Thomas Franke, Nadine Rauh, Josef F. Krems 2017 - ~~"Perceived Performance of Top Retail Webpages In the Wild: Insights from Large-scale Crowdsourcing of Above-the-Fold QoE" Gao, Dey, Ahammad 2017~~ - ~~"PAIN: A Passive Web Speed Indicator for ISPs" Marco Mellia, Idilio Drago, Martino Trevisan 2017~~ - ~~[[ http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7888987 | "Reordering Webpage Objects for Optimizing Quality-of-Experience" Li, Zhao, Min, Duan, Ni, Zhao 2017 ]]~~ - ~~[[ https://www.usenix.org/system/files/conference/nsdi17/nsdi17-kelton.pdf | "Improving User Perceived Page Load Times Using Gaze" Kelton, Ryoo, Balasubramanian, Das 2017 ]]~~ - ~~[[ http://web.mit.edu/ravinet/vesper_nsdi18.pdf | "Vesper: Measuring Time-to-Interactivity for Modern Web Pages" Netravali, Nathan, Mickens, Balakrishnan 2017 ]]~~ - ~~[[ https://pdfs.semanticscholar.org/3a8e/2eb20f987e518458386dcd44fd8d11df936d.pdf | "Narrowing the gap between QoS metrics and Web QoE using Above-the-fold metrics" Diego Neves da Hora, Alemnew Sheferaw Asrese, Vassilis Christophides, Renata Teixeira, Dario Rossi 2018 ]]~~ It would be nice to put together a literature review at the end of this, summarizing the different papers read and drawing connections between them.

We tend to point to guidelines like RAIL in performance discussions, but the figures put forward in these guidelines rarely, if ever, reference the underlying research. There might not be any data or study that actually prove the existence of the thresholds described or their values. We should formally review what's been done to make up our own opinion about performance speed guidelines, or even come up with guidelines of our own, or create new studies that would fill the gaps. It would be interesting to specifically look for studies based on real data/research that answers the following questions: - What's the speed threshold for what feels instantaneous? In what context? - What's the speed threshold for something so slow that it becomes frustrating to the user? In what context? - In what contexts has "faster is better" been demonstrated? Is there any other context where slower has been proven to be better? - What's the distribution of performance perception between different people? We should be particularly wary of studies that take mental speed/time thresholds for granted. I've seen it to be quite common that previous research papers are referenced on that matter, but the original papers didn't base those thresholds on any research, they were actually part of their unconfirmed hypothesis. Some of the more recent papers are quite critical of that, observing that those "magic numbers" keep getting propagated but have not necessarily been proven. Below is a working list of research papers of interest, that keeps expanding as we explore them and their citations. The starting point was [[ https://www.smashingmagazine.com/2015/10/rail-user-centric-model-performance/ | the list quoted in the RAIL announcement ]], as well as searches for perceived performance keywords in google scholar. - ~~[[ http://theixdlibrary.com/pdf/Miller1968.pdf | Miller, R.B., 1968, Response time in man-computer conversational transaction. Proceedings of AFIPS Fall Joint Computer Conference, 33, 267-277. ]]~~ - Rule, S.J., Curtis, D.W. and Markley, R.P., 1970, Input and output transformations from magnitude estimation. Journal of Experimental Psychology, 86(1), 343-349. - Hornik, J., 1986, Subjective vs. objective time measures: a note on the perception of time in consumer behavior. Journal of Consumer Research, 11, 615-618. - Guynes, J.L., 1988, Impact of system response time on state anxiety. Communications of the ACM, 31(3), 342-347. - [[ https://www.researchgate.net/profile/Richard_Larson5/publication/304582002_Prescription_for_the_Waiting_in_Line_Blues_Entertain_Enlighten_Engage/links/5774627708aeb9427e2422c4/Prescription-for-the-Waiting-in-Line-Blues-Entertain-Enlighten-Engage.pdf | Katz, K., Larson, B. and Larson, R., 1991, Prescription for the waiting in line blues: entertain, enlighten and engage. Sloan Management Review, 32(2), 44-53. ]] - ~~[[ http://www.useit.com/papers/responsetime.html | Nielsen, J., 1993, Response times: the three important limits. Excerpt from Chapter 5 of Usability Engineering by Jakob Nielsen, Academic Press, 1993. ]]~~ - Taylor, S., 1994, Waiting for service: the relationship between delays and evaluations of service. Journal of Marketing, 58(2), 56-69. - Hui, M.K. and Tse, D., 1996, What to tell consumers in waits of different lengths: an integrative model of service evaluation. Journal of Marketing, 60(2), 81-90. - Hui, M.K. and Zhou, L., 1996, How does waiting duration information influence customers’ reactions to waiting for services? Journal of Applied Social Psychology, 26(19), 1702-1717. - Ramsay, J., Barbesi, A. and Preece, J., 1998, A psychological investigation of long retrieval times on the world wide web. Interacting with Computers, 10(1), 77-86. - Dellaert, B.G.C. and Kahn, B.E., 1999, How tolerable is delay? Consumers’ evaluations of Internet Web sites after waiting. Journal of Interactive Marketing, 13(1), 41-54. - [[ http://usabilitynews.org/how-long-is-too-long-to-wait-for-a-website-to-load/ | Selvidge, P., 1999, How long is too long for a website to load? Usability News, 1(2). ]] - Jacko, J.A., Sears, A. and Borella, M.S., 2000, The effect of network delay and media on user perceptions of web resources. Behaviour and Information Technology, 19(6), 427-439. - [[ http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.99.2770 | Hoxmeier, J.A. and DiCesare, C., 2000, System response time and user satisfaction: an experimental study of browser-based applications. Proceedings of the Americas Conference on Information Systems, 10-13 August 2000 (Long Beach, California: Association for Information Systems), pp. 140-145. ]] - Weinberg, B.D., 2000, Don’t keep your Internet customers waiting too long at the (virtual) front door. Journal of Interactive Marketing, 14(10), 30-38. - [[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.907.7268&rep=rep1&type=pdf | Rose, G.M. and Straub, D.W., 2001, The effect of download time on consumer attitude toward the e-service retailer. e-Service Journal, 1, 55-76. ]] - [[ http://www.mediajournal.org/index.php/jmm/article/view/167/50 | Rose, G.M., Lees, J. and Meuter, M.L., 2001, A refined view of download time impacts on e-consumer attitudes and patronage intentions toward e-retailers. International Journal of Media Management, 3, 105-111. ]] - ~~Zona Research, 2001, The need for speed II. Zona Market Bulletin, 5, April 2001.~~ (can't find it) - Antonides, G., Verhoef, P.C., van Aalst, M., 2002, Consumer perception and evaluation of waiting time: a field experiment. Journal of Consumer Psychology, 12(3), 193-202. - [[ http://melody.syr.edu/hci/pre_icis02_wksp/pres/11_Galletta.ppt | Galletta, D., Henry, R., McCoy, S. and Polak, P., 2002, Web site delays: how slow can you go?” Presented at the First Annual Pre-ICIS Workshop on HCI Research in MIS, 14 December 2002 forthcoming in Journal of Association for Information Systems. ]] - ~~[[ http://usabilitynews.org/examining-tolerance-for-online-delays/ | Selvidge, P., 2003, Examining tolerance for online delays. Usability News, 5(1). ]]~~ - Rose, G.M., Evaristo, R. and Straub, D., 2003, Culture and consumer responses to web download time: a four-continent study of mono and polychronism. IEEE Transactions on Engineering Management, 50(1), 31-44. - ~~[[ http://sighci.org/uploads/published_papers/bit04/BIT_Nah.pdf | Nah, Fiona, "A Study on Tolerable Waiting Time: How Long Are Web Users Willing to Wait?" (2003). AMCIS 2003 Proceedings. 285. ]]~~ - ~~[[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.155&rep=rep1&type=pdf | 2005: “Interaction in 4-Second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI” Antti Oulasvirta, Sakari Tamminen, Virpi Roto, and Jaana Kuorelahti, Interruptions in Human Computer Interaction ]]~~ - "The influence of cognitive load, physical load and system response time on user productivity and satisfaction with computer systems" Dabrowski, James R. 2010 - [[ http://isr.cmu.edu/doc/tolia06-ieee.pdf | 2006: “Quantifying Interactive User Experience on Thin Clients”, Niraj Tolia, David G. Andersen, and M. Satyanarayanan, The Internet Suspend/Resume Project, Carnegie Mellon ]] - ~~[[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.366.1170&rep=rep1&type=pdf | 2012: “Characterizing Web Use on Smartphones”, Chad C. Tossell, Philip Kortum, Ahmad Rahmati, Clayton Shepard, Lin Zhong, Conference on Human Factors in Computing Systems 2012 ]]~~ - ~~[[ https://link.springer.com/chapter/10.1007%2F978-3-642-23771-3_42 | Kaaresoja T., Hoggan E., Anttila E. (2011) Playing with Tactile Feedback Latency in Touchscreen Interaction: Two Approaches. In: Campos P., Graham N., Jorge J., Nunes N., Palanque P., Winckler M. (eds) Human-Computer Interaction – INTERACT 2011. INTERACT 2011. ]]~~ - ~~[[ http://www.ingmarverheij.com/wp-content/uploads/downloads/2011/09/Whitepaper-Quantifying-Perceived-Performance-v1.0.pdf | "Quantify Perceived Performance" Ingmar Verheij 2011 ]]~~ - ~~[[ https://pdfs.semanticscholar.org/5c5c/7a2b067cddea7dedf3f66dce8b3f93558b72.pdf | "Improving the Human–Computer Dialogue With Increased Temporal Predictability" Florian Weber, Carola Haering, Roland Thomaschke 2013 ]]~~ - ~~[[ https://homes.cs.washington.edu/~arvind/papers/wprof.pdf | "Demystifying Page Load Performance with WProf" Wang, Balasubramanian, Krishnamurthy, Wetherall 2013 ]]~~ - "A methodology for the evaluation of high response time on E-commerce users and sales" Nicolas Poggi, David Carrera, Ricard Gavaldà, Eduard Ayguadé, Jordi Torres 2014 - ~~[[ http://www.dmax.org.uk/uploads/publications/iiix2014cost_performance_interaction.pdf | "Stuck in traffic: how temporal delays affect search behaviour" David Maxwell, Leif Azzopardi 2014 ]]~~ - ~~[[ https://www.diva-portal.org/smash/get/diva2:832423/FULLTEXT01.pdf | "The Impact of Waiting Time Distributions on QoE of Task-Based Web Browsing Sessions" Islam Nazrul, Elepe Vijaya John David 2014 ]]~~ - "Perception of Delay and Attitude toward Feedback Display: An Exploration into Downloaders’ Demographics" Chatpong Tangmanee, Pawarat Nontasil 2014 - [[ https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chow.pdf | "The Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services" Chow, Meisner, Flinn, Peek, Wenisch 2014 ]] - ~~[[ https://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/43224.pdf | "Towards Better Measurement of Attention and Satisfaction in Mobile Search" Lagun, Hsieh, Webster, Navalpakkam 2014]]~~ - ~~[[ https://research.spec.org/icpe_proceedings/2015/icpe/p245.pdf | "Defining Standards for Web Page Performance in Business Applications" Garret Rempel 2015 ]]~~ - ~~"User-Acceptance of Latency in Touch Interactions" Walter Ritter, Guido Kempter, Tobias Werner 2015~~ - ~~[[ https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-butkiewicz.pdf | "Klotski: Reprioritizing Web Content to Improve User Experience on Mobile Devices" Butkiewicz, Wang, Wu, Madhyastha, Sekar 2015 ]]~~ - "Predicting web page performance level based on web page characteristics" Zhou, Zhang, Zhou, Li 2015 - ~~[[ https://perso.telecom-paristech.fr/~drossi/paper/rossi16internet-qoe.pdf | "Measuring the Quality of Experience of Web users" Bocchi, De Cicco, Rossi 2016 ]]~~ - ~~[[ https://arxiv.org/pdf/1603.02293.pdf | "What slows you down? Your network or your device?" Steiner, Gao 2016 ]]~~ - ~~[[ http://sattose.wdfiles.com/local--files/2016:alltalks/SATTOSE2016_paper_13.pdf | "Critical CSS Rules — Decreasing time to first render by inlining CSS rules for over-the-fold elements" Jovanovski, Zaytsev 2016 ]]~~ - ~~[[ https://www.mccormick.northwestern.edu/eecs/documents/tech-reports/2016/2016-10-oak-user-targeted-web-performance.pdf | "Oak: User-Targeted Web Performance" Flores, Wenzel, Kuzmanovic 2016 ]]~~ - ~~[[ https://davidtnaylor.com/eyeorg.pdf | "EYEORG: A Platform For Crowdsourcing Web Quality Of Experience Measurements" Varvello, Blackburn, Naylor, Papagiannaki, 2016 ]]~~ - ~~[[ http://fernando.diaz.nyc/WhiteTOIS2016.pdf | "Search Result Prefetching on Desktop and Mobile" Ryen W. White, Fernando Diaz, Qi Guo, 2017 ]]~~ - ~~"The duration perception of loading applications in smartphone: Effects of different loading types" Wenguo Zhao, Yan Ge, Weina Qu, Kan Zhang, Xianghong Sun 2017~~ - ~~[[ https://perso.telecom-paristech.fr/drossi/paper/rossi17pam.pdf | "The Web, the Users, and the MOS: Influence of HTTP/2 on User Experience" Enrico Bocchi, Luca De Cicco, Marco Mellia, Dario Rossi, 2017 ]]~~ - "Are 100 ms Fast Enough? Characterizing Latency Perception Thresholds in Mouse-Based Interaction" Valentin Forch, Thomas Franke, Nadine Rauh, Josef F. Krems 2017 - ~~"Perceived Performance of Top Retail Webpages In the Wild: Insights from Large-scale Crowdsourcing of Above-the-Fold QoE" Gao, Dey, Ahammad 2017~~ - ~~"PAIN: A Passive Web Speed Indicator for ISPs" Marco Mellia, Idilio Drago, Martino Trevisan 2017~~ - ~~[[ http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7888987 | "Reordering Webpage Objects for Optimizing Quality-of-Experience" Li, Zhao, Min, Duan, Ni, Zhao 2017 ]]~~ - ~~[[ https://www.usenix.org/system/files/conference/nsdi17/nsdi17-kelton.pdf | "Improving User Perceived Page Load Times Using Gaze" Kelton, Ryoo, Balasubramanian, Das 2017 ]]~~ - ~~[[ http://web.mit.edu/ravinet/vesper_nsdi18.pdf | "Vesper: Measuring Time-to-Interactivity for Modern Web Pages" Netravali, Nathan, Mickens, Balakrishnan 2017 ]]~~ - ~~[[ https://pdfs.semanticscholar.org/3a8e/2eb20f987e518458386dcd44fd8d11df936d.pdf | "Narrowing the gap between QoS metrics and Web QoE using Above-the-fold metrics" Diego Neves da Hora, Alemnew Sheferaw Asrese, Vassilis Christophides, Renata Teixeira, Dario Rossi 2018 ]]~~ It would be nice to put together a literature review at the end of this, summarizing the different papers read and drawing connections between them.

We tend to point to guidelines like RAIL in performance discussions, but the figures put forward in these guidelines rarely, if ever, reference the underlying research. There might not be any data or study that actually prove the existence of the thresholds described or their values. We should formally review what's been done to make up our own opinion about performance speed guidelines, or even come up with guidelines of our own, or create new studies that would fill the gaps. It would be interesting to specifically look for studies based on real data/research that answers the following questions: - What's the speed threshold for what feels instantaneous? In what context? - What's the speed threshold for something so slow that it becomes frustrating to the user? In what context? - In what contexts has "faster is better" been demonstrated? Is there any other context where slower has been proven to be better? - What's the distribution of performance perception between different people? We should be particularly wary of studies that take mental speed/time thresholds for granted. I've seen it to be quite common that previous research papers are referenced on that matter, but the original papers didn't base those thresholds on any research, they were actually part of their unconfirmed hypothesis. Some of the more recent papers are quite critical of that, observing that those "magic numbers" keep getting propagated but have not necessarily been proven. Below is a working list of research papers of interest, that keeps expanding as we explore them and their citations. The starting point was [[ https://www.smashingmagazine.com/2015/10/rail-user-centric-model-performance/ | the list quoted in the RAIL announcement ]], as well as searches for perceived performance keywords in google scholar. - ~~[[ http://theixdlibrary.com/pdf/Miller1968.pdf | Miller, R.B., 1968, Response time in man-computer conversational transaction. Proceedings of AFIPS Fall Joint Computer Conference, 33, 267-277. ]]~~ - Rule, S.J., Curtis, D.W. and Markley, R.P., 1970, Input and output transformations from magnitude estimation. Journal of Experimental Psychology, 86(1), 343-349. - Hornik, J., 1986, Subjective vs. objective time measures: a note on the perception of time in consumer behavior. Journal of Consumer Research, 11, 615-618. - Guynes, J.L., 1988, Impact of system response time on state anxiety. Communications of the ACM, 31(3), 342-347. - [[ https://www.researchgate.net/profile/Richard_Larson5/publication/304582002_Prescription_for_the_Waiting_in_Line_Blues_Entertain_Enlighten_Engage/links/5774627708aeb9427e2422c4/Prescription-for-the-Waiting-in-Line-Blues-Entertain-Enlighten-Engage.pdf | Katz, K., Larson, B. and Larson, R., 1991, Prescription for the waiting in line blues: entertain, enlighten and engage. Sloan Management Review, 32(2), 44-53. ]] - ~~[[ http://www.useit.com/papers/responsetime.html | Nielsen, J., 1993, Response times: the three important limits. Excerpt from Chapter 5 of Usability Engineering by Jakob Nielsen, Academic Press, 1993. ]]~~ - Taylor, S., 1994, Waiting for service: the relationship between delays and evaluations of service. Journal of Marketing, 58(2), 56-69. - Hui, M.K. and Tse, D., 1996, What to tell consumers in waits of different lengths: an integrative model of service evaluation. Journal of Marketing, 60(2), 81-90. - Hui, M.K. and Zhou, L., 1996, How does waiting duration information influence customers’ reactions to waiting for services? Journal of Applied Social Psychology, 26(19), 1702-1717. - Ramsay, J., Barbesi, A. and Preece, J., 1998, A psychological investigation of long retrieval times on the world wide web. Interacting with Computers, 10(1), 77-86. - Dellaert, B.G.C. and Kahn, B.E., 1999, How tolerable is delay? Consumers’ evaluations of Internet Web sites after waiting. Journal of Interactive Marketing, 13(1), 41-54. - [[ http://usabilitynews.org/how-long-is-too-long-to-wait-for-a-website-to-load/ | Selvidge, P., 1999, How long is too long for a website to load? Usability News, 1(2). ]] - Jacko, J.A., Sears, A. and Borella, M.S., 2000, The effect of network delay and media on user perceptions of web resources. Behaviour and Information Technology, 19(6), 427-439. - [[ http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.99.2770 | Hoxmeier, J.A. and DiCesare, C., 2000, System response time and user satisfaction: an experimental study of browser-based applications. Proceedings of the Americas Conference on Information Systems, 10-13 August 2000 (Long Beach, California: Association for Information Systems), pp. 140-145. ]] - Weinberg, B.D., 2000, Don’t keep your Internet customers waiting too long at the (virtual) front door. Journal of Interactive Marketing, 14(10), 30-38. - [[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.907.7268&rep=rep1&type=pdf | Rose, G.M. and Straub, D.W., 2001, The effect of download time on consumer attitude toward the e-service retailer. e-Service Journal, 1, 55-76. ]] - [[ http://www.mediajournal.org/index.php/jmm/article/view/167/50 | Rose, G.M., Lees, J. and Meuter, M.L., 2001, A refined view of download time impacts on e-consumer attitudes and patronage intentions toward e-retailers. International Journal of Media Management, 3, 105-111. ]] - ~~Zona Research, 2001, The need for speed II. Zona Market Bulletin, 5, April 2001.~~ (can't find it) - Antonides, G., Verhoef, P.C., van Aalst, M., 2002, Consumer perception and evaluation of waiting time: a field experiment. Journal of Consumer Psychology, 12(3), 193-202. - [[ http://melody.syr.edu/hci/pre_icis02_wksp/pres/11_Galletta.ppt | Galletta, D., Henry, R., McCoy, S. and Polak, P., 2002, Web site delays: how slow can you go?” Presented at the First Annual Pre-ICIS Workshop on HCI Research in MIS, 14 December 2002 forthcoming in Journal of Association for Information Systems. ]] - ~~[[ http://usabilitynews.org/examining-tolerance-for-online-delays/ | Selvidge, P., 2003, Examining tolerance for online delays. Usability News, 5(1). ]]~~ - Rose, G.M., Evaristo, R. and Straub, D., 2003, Culture and consumer responses to web download time: a four-continent study of mono and polychronism. IEEE Transactions on Engineering Management, 50(1), 31-44. - ~~[[ http://sighci.org/uploads/published_papers/bit04/BIT_Nah.pdf | Nah, Fiona, "A Study on Tolerable Waiting Time: How Long Are Web Users Willing to Wait?" (2003). AMCIS 2003 Proceedings. 285. ]]~~ - ~~[[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.155&rep=rep1&type=pdf | 2005: “Interaction in 4-Second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI” Antti Oulasvirta, Sakari Tamminen, Virpi Roto, and Jaana Kuorelahti, Interruptions in Human Computer Interaction ]]~~ - [[ http://ianspiro.com/chi2008.pdf | "Eye-mouse coordination patterns on web search results pages" Rodden, Fu, Aula, Spiro 2008 ]] - "The influence of cognitive load, physical load and system response time on user productivity and satisfaction with computer systems" Dabrowski, James R. 2010 - "WebProphet: automating performance prediction for web services" Li, Zhang, Zhu, Chen, Greenberg, Wang 2010 - [[ http://isr.cmu.edu/doc/tolia06-ieee.pdf | 2006: “Quantifying Interactive User Experience on Thin Clients”, Niraj Tolia, David G. Andersen, and M. Satyanarayanan, The Internet Suspend/Resume Project, Carnegie Mellon ]] - ~~[[ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.366.1170&rep=rep1&type=pdf | 2012: “Characterizing Web Use on Smartphones”, Chad C. Tossell, Philip Kortum, Ahmad Rahmati, Clayton Shepard, Lin Zhong, Conference on Human Factors in Computing Systems 2012 ]]~~ - ~~[[ https://link.springer.com/chapter/10.1007%2F978-3-642-23771-3_42 | Kaaresoja T., Hoggan E., Anttila E. (2011) Playing with Tactile Feedback Latency in Touchscreen Interaction: Two Approaches. In: Campos P., Graham N., Jorge J., Nunes N., Palanque P., Winckler M. (eds) Human-Computer Interaction – INTERACT 2011. INTERACT 2011. ]]~~ - ~~[[ http://www.ingmarverheij.com/wp-content/uploads/downloads/2011/09/Whitepaper-Quantifying-Perceived-Performance-v1.0.pdf | "Quantify Perceived Performance" Ingmar Verheij 2011 ]]~~ - ~~[[ https://pdfs.semanticscholar.org/5c5c/7a2b067cddea7dedf3f66dce8b3f93558b72.pdf | "Improving the Human–Computer Dialogue With Increased Temporal Predictability" Florian Weber, Carola Haering, Roland Thomaschke 2013 ]]~~ - ~~[[ https://homes.cs.washington.edu/~arvind/papers/wprof.pdf | "Demystifying Page Load Performance with WProf" Wang, Balasubramanian, Krishnamurthy, Wetherall 2013 ]]~~ - "A methodology for the evaluation of high response time on E-commerce users and sales" Nicolas Poggi, David Carrera, Ricard Gavaldà, Eduard Ayguadé, Jordi Torres 2014 - ~~[[ http://www.dmax.org.uk/uploads/publications/iiix2014cost_performance_interaction.pdf | "Stuck in traffic: how temporal delays affect search behaviour" David Maxwell, Leif Azzopardi 2014 ]]~~ - ~~[[ https://www.diva-portal.org/smash/get/diva2:832423/FULLTEXT01.pdf | "The Impact of Waiting Time Distributions on QoE of Task-Based Web Browsing Sessions" Islam Nazrul, Elepe Vijaya John David 2014 ]]~~ - "Perception of Delay and Attitude toward Feedback Display: An Exploration into Downloaders’ Demographics" Chatpong Tangmanee, Pawarat Nontasil 2014 - [[ https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chow.pdf | "The Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services" Chow, Meisner, Flinn, Peek, Wenisch 2014 ]] - ~~[[ https://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/43224.pdf | "Towards Better Measurement of Attention and Satisfaction in Mobile Search" Lagun, Hsieh, Webster, Navalpakkam 2014]]~~ - ~~[[ https://research.spec.org/icpe_proceedings/2015/icpe/p245.pdf | "Defining Standards for Web Page Performance in Business Applications" Garret Rempel 2015 ]]~~ - ~~"User-Acceptance of Latency in Touch Interactions" Walter Ritter, Guido Kempter, Tobias Werner 2015~~ - ~~[[ https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-butkiewicz.pdf | "Klotski: Reprioritizing Web Content to Improve User Experience on Mobile Devices" Butkiewicz, Wang, Wu, Madhyastha, Sekar 2015 ]]~~ - [[ https://dspace.mit.edu/bitstream/handle/1721.1/100162/928920895-MIT.pdf?sequence=1 | "DepTracker: Tracking JavaScript Dependencies on the Web" Goyal 2015 ]] - "Predicting web page performance level based on web page characteristics" Zhou, Zhang, Zhou, Li 2015 - ~~[[ https://perso.telecom-paristech.fr/~drossi/paper/rossi16internet-qoe.pdf | "Measuring the Quality of Experience of Web users" Bocchi, De Cicco, Rossi 2016 ]]~~ - [[ https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-netravali.pdf | "Polaris: Faster Page Loads Using Fine-grained Dependency Tracking" Netravali, Goyal, Mickens, Balakrishnan 2016 ]] - [[ https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-wang-xiao-sophia.pdf | "Speeding up Web Page Loads with Shandian" Wang, Krishnamurthy, Wetherall 2016 ]] - [[ https://arxiv.org/pdf/1602.03273.pdf | "YTrace: End-to-end Performance Diagnosis in Large Cloud and Content Providers" Kanuparthy, Dai, Pathak, Samal, Benson, Ghasemi, Narayan 2016 ]] - ~~[[ https://arxiv.org/pdf/1603.02293.pdf | "What slows you down? Your network or your device?" Steiner, Gao 2016 ]]~~ - ~~[[ http://sattose.wdfiles.com/local--files/2016:alltalks/SATTOSE2016_paper_13.pdf | "Critical CSS Rules — Decreasing time to first render by inlining CSS rules for over-the-fold elements" Jovanovski, Zaytsev 2016 ]]~~ - ~~[[ https://www.mccormick.northwestern.edu/eecs/documents/tech-reports/2016/2016-10-oak-user-targeted-web-performance.pdf | "Oak: User-Targeted Web Performance" Flores, Wenzel, Kuzmanovic 2016 ]]~~ - ~~[[ https://davidtnaylor.com/eyeorg.pdf | "EYEORG: A Platform For Crowdsourcing Web Quality Of Experience Measurements" Varvello, Blackburn, Naylor, Papagiannaki, 2016 ]]~~ - [[ https://www.usenix.org/system/files/conference/atc16/atc16-paper-vesuna.pdf | "Caching Doesn’t Improve Mobile Web Performance (Much)" Vesuna, Scott, Buettner, Piatek, Krishnamurthy, Shenker 2016 ]] - ~~[[ http://fernando.diaz.nyc/WhiteTOIS2016.pdf | "Search Result Prefetching on Desktop and Mobile" Ryen W. White, Fernando Diaz, Qi Guo, 2017 ]]~~ - ~~"The duration perception of loading applications in smartphone: Effects of different loading types" Wenguo Zhao, Yan Ge, Weina Qu, Kan Zhang, Xianghong Sun 2017~~ - ~~[[ https://perso.telecom-paristech.fr/drossi/paper/rossi17pam.pdf | "The Web, the Users, and the MOS: Influence of HTTP/2 on User Experience" Enrico Bocchi, Luca De Cicco, Marco Mellia, Dario Rossi, 2017 ]]~~ - "Are 100 ms Fast Enough? Characterizing Latency Perception Thresholds in Mouse-Based Interaction" Valentin Forch, Thomas Franke, Nadine Rauh, Josef F. Krems 2017 - ~~"Perceived Performance of Top Retail Webpages In the Wild: Insights from Large-scale Crowdsourcing of Above-the-Fold QoE" Gao, Dey, Ahammad 2017~~ - ~~"PAIN: A Passive Web Speed Indicator for ISPs" Marco Mellia, Idilio Drago, Martino Trevisan 2017~~ - ~~[[ http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7888987 | "Reordering Webpage Objects for Optimizing Quality-of-Experience" Li, Zhao, Min, Duan, Ni, Zhao 2017 ]]~~ - ~~[[ https://www.usenix.org/system/files/conference/nsdi17/nsdi17-kelton.pdf | "Improving User Perceived Page Load Times Using Gaze" Kelton, Ryoo, Balasubramanian, Das 2017 ]]~~ - ~~[[ http://web.mit.edu/ravinet/vesper_nsdi18.pdf | "Vesper: Measuring Time-to-Interactivity for Modern Web Pages" Netravali, Nathan, Mickens, Balakrishnan 2017 ]]~~ - ~~[[ https://pdfs.semanticscholar.org/3a8e/2eb20f987e518458386dcd44fd8d11df936d.pdf | "Narrowing the gap between QoS metrics and Web QoE using Above-the-fold metrics" Diego Neves da Hora, Alemnew Sheferaw Asrese, Vassilis Christophides, Renata Teixeira, Dario Rossi 2018 ]]~~ It would be nice to put together a literature review at the end of this, summarizing the different papers read and drawing connections between them.