Page MenuHomePhabricator

Establish a user testing framework for perceived speed
Closed, DeclinedPublic


There are a number of potential improvements to Media Viewer that could be considered in order to improve the speed perception experienced by users. However, there is no good way to verify that our intuition about these effects is correct. We need a solid way of user-testing 2 or more options in order to determine which one universally "feels fastest" to users, for any given product. Ideally it should be easy to set up, as we might have many combinations to test and we'll probably want to have a large sample of users to make the results meaningful.

Here are examples of hypothetical changes that may improve or worsen perceived speed on Media Viewer, which have been floating around as ideas or changesets but not merged because there's no good way to tell if they're really improvements:

  • Getting rid of the animation from the blurred blown-up placeholder to the final image. While at first glance it might seem obvious that no animation is better (after all, we'd be displaying the visual information as soon as it's available), the jarring change might bring more attention to the waiting time that just happened.
  • Switching from a blur effect to a pixellation effect for the placeholder.
  • Having no effect at all and displaying the blown-up placeholder as-is, JPG artifacts and all.
  • Not displaying any placeholder (leaving the screen black, or the previous image if any)
  • Displaying a spinner on the thumbnail while Media Viewer's JS is loading.
  • Displaying a black screen with minimal UI, including an indefinite progress bar while Media Viewer's JS is loading.
  • Displaying an indefinite progress bar on the thumbnail followed by a real progress bar, until the actual image is loaded at which point we display the Media Viewer UI, including the sharp image.
  • Displaying an indefinite progress bar instead of the real progress bar for images. The organic jumping of the real progress bar due to network conditions might feel distracting or frustrating.
  • Not displaying the progress bar at all on fast connections. If the progress bar appear for less than a second/X milliseconds, it might be a distraction that makes the interface feel slow, while displaying the progress bar might be a useful placeholder when it takes several seconds to load the image. I.e. depending on how long it's on the screen for, the progress bar might be a good or a bad. An easy way to test this is to make the progress bar appear only if the loading has been taking more than X milliseconds.
  • Adapting the preloading distance depending on how fast the person is prev/nexting images.
  • Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI
  • Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI minus the progress bar
  • Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI minus the blurry placeholder
  • Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI minus the blurry placeholder and the progress bar

Event Timeline

Gilles raised the priority of this task from to Needs Triage.
Gilles updated the task description. (Show Details)
Gilles added subscribers: Gilles, Tgr.

I've spent some time looking for research or material on the matter and unfortunately all I could find was subjective advice about what feels fastest to people, with no data to back it up. And several real-life examples which were effective, but that do not translate to UX at all. Like for example the classic scheme of putting mirrors in places where people wait (elevators, queues) to make the wait feel shorter, or making people artificially walk for X minutes inside airport terminals so that they complain less about waiting for their luggage. Both of which would suggest that displaying something is better than displaying nothing during a waiting period, but as the examples show above, it doesn't help with deciding *what* to show.

There is some research on the effect of progress bars on perceived performance, like this (Progress bars with animated ribbing that move backwards in a decelerating manner proved to have the strongest effect. In a final experiment, we measured the effect of this particular progress bar design and showed that it reduces the perceived duration among our participants by 11%.) or this (users seem to have a strong aversion to pauses especially towards the end of an operation ... progress can be downplayed in the beginning and accelerated towards the end, providing a sense of a rapid conclusion that is highly favored by users in our experiment, ie. use an easing function that smoothes out jerkiness and maps the same amount of progress to a larger area towards the end of the bar, or use an indeterminate progress bar with accelerating ribbon animation).

The closest parallel to MMV's loading effect is probably the debate about interlaced vs. normal image loading (interesting in its own right as well, as generating interlaced thumbnails would be an easy tweak), but there is surprisingly little research about the perceived effects. The only one I could find is this slightly advertisey piece which claims large penalties on interlaced loading (but given that it's focused on pushing the researchers' own proprietary solution and uses rather obscure methodology, I would take it with a grain of salt).

I've given this more thought and I think that things can be simplified by looking at the bigger picture. Perceived speed is only one dimension of user satisfaction, other dimensions are for example how visually pleasing the feature is, how smooth it feels, etc. I think it's very difficult to measure those individual feelings about the product. However, we can easily check these changes against success metrics. In particular our main success metric, which was used to compare Media Viewer to the file page: image view counts. If people like a slightly different version of Media Viewer better, over all their satisfaction criteria, they should be using it more. Looking at the problem that way actually makes pitting Media Viewer X against Media Viewer Y very simple.

gerritbot subscribed.

Change 186158 had a related patch set uploaded (by Gilles):
[WIP] Variant logging