Establish a user testing framework for perceived speed
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• Gilles
	Jan 19 2015, 3:12 PM

Description

There are a number of potential improvements to Media Viewer that could be considered in order to improve the speed perception experienced by users. However, there is no good way to verify that our intuition about these effects is correct. We need a solid way of user-testing 2 or more options in order to determine which one universally "feels fastest" to users, for any given product. Ideally it should be easy to set up, as we might have many combinations to test and we'll probably want to have a large sample of users to make the results meaningful.

Here are examples of hypothetical changes that may improve or worsen perceived speed on Media Viewer, which have been floating around as ideas or changesets but not merged because there's no good way to tell if they're really improvements:

Getting rid of the animation from the blurred blown-up placeholder to the final image. While at first glance it might seem obvious that no animation is better (after all, we'd be displaying the visual information as soon as it's available), the jarring change might bring more attention to the waiting time that just happened.
Switching from a blur effect to a pixellation effect for the placeholder.
Having no effect at all and displaying the blown-up placeholder as-is, JPG artifacts and all.
Not displaying any placeholder (leaving the screen black, or the previous image if any)
Displaying a spinner on the thumbnail while Media Viewer's JS is loading.
Displaying a black screen with minimal UI, including an indefinite progress bar while Media Viewer's JS is loading.
Displaying an indefinite progress bar on the thumbnail followed by a real progress bar, until the actual image is loaded at which point we display the Media Viewer UI, including the sharp image.
Displaying an indefinite progress bar instead of the real progress bar for images. The organic jumping of the real progress bar due to network conditions might feel distracting or frustrating.
Not displaying the progress bar at all on fast connections. If the progress bar appear for less than a second/X milliseconds, it might be a distraction that makes the interface feel slow, while displaying the progress bar might be a useful placeholder when it takes several seconds to load the image. I.e. depending on how long it's on the screen for, the progress bar might be a good or a bad. An easy way to test this is to make the progress bar appear only if the loading has been taking more than X milliseconds.
Adapting the preloading distance depending on how fast the person is prev/nexting images.
Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI
Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI minus the progress bar
Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI minus the blurry placeholder
Displaying the progressive vertical appearance of the image loaded so far, on top of the existing UI minus the blurry placeholder and the progress bar

Details

	Subject	Repo	Branch	Lines +/-
	Variant logging	mediawiki/extensions/MultimediaViewer	master	+170 -9

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T158181 Aim for workflow equivalence for MediaWiki on desktop and mobile web
Declined	None	T144571 Mobile Media Viewer violates user settings
Open	Jdlrobson	T65504 Consolidate code for MediaViewer across desktop and mobile
Open	None	T71609 Test if replacing the blur effect with pixelation makes loading feel faster
Declined	None	T87184 Establish a user testing framework for perceived speed

Event Timeline

• Gilles created this task.Jan 19 2015, 3:12 PM

• Gilles raised the priority of this task from to Needs Triage.

• Gilles updated the task description. (Show Details)

• Gilles added projects: Wikimedia-Design, Design-Research.

• Gilles added subscribers: • Gilles, Tgr.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 19 2015, 3:12 PM

I've spent some time looking for research or material on the matter and unfortunately all I could find was subjective advice about what feels fastest to people, with no data to back it up. And several real-life examples which were effective, but that do not translate to UX at all. Like for example the classic scheme of putting mirrors in places where people wait (elevators, queues) to make the wait feel shorter, or making people artificially walk for X minutes inside airport terminals so that they complain less about waiting for their luggage. Both of which would suggest that displaying something is better than displaying nothing during a waiting period, but as the examples show above, it doesn't help with deciding *what* to show.

There is some research on the effect of progress bars on perceived performance, like this (Progress bars with animated ribbing that move backwards in a decelerating manner proved to have the strongest effect. In a final experiment, we measured the effect of this particular progress bar design and showed that it reduces the perceived duration among our participants by 11%.) or this (users seem to have a strong aversion to pauses especially towards the end of an operation ... progress can be downplayed in the beginning and accelerated towards the end, providing a sense of a rapid conclusion that is highly favored by users in our experiment, ie. use an easing function that smoothes out jerkiness and maps the same amount of progress to a larger area towards the end of the bar, or use an indeterminate progress bar with accelerating ribbon animation).

The closest parallel to MMV's loading effect is probably the debate about interlaced vs. normal image loading (interesting in its own right as well, as generating interlaced thumbnails would be an easy tweak), but there is surprisingly little research about the perceived effects. The only one I could find is this slightly advertisey piece which claims large penalties on interlaced loading (but given that it's focused on pushing the researchers' own proprietary solution and uses rather obscure methodology, I would take it with a grain of salt).

Tgr added a parent task: T71609: Test if replacing the blur effect with pixelation makes loading feel faster.Jan 19 2015, 10:05 PM

Aklapper added a project: Performance Issue.Jan 20 2015, 10:01 PM

I've given this more thought and I think that things can be simplified by looking at the bigger picture. Perceived speed is only one dimension of user satisfaction, other dimensions are for example how visually pleasing the feature is, how smooth it feels, etc. I think it's very difficult to measure those individual feelings about the product. However, we can easily check these changes against success metrics. In particular our main success metric, which was used to compare Media Viewer to the file page: image view counts. If people like a slightly different version of Media Viewer better, over all their satisfaction criteria, they should be using it more. Looking at the problem that way actually makes pitting Media Viewer X against Media Viewer Y very simple.