Page MenuHomePhabricator

Usability test “Add custom tags“ (2020-03-18)
Closed, ResolvedPublic

Description

TL;DR / action items

  • 👎4/5 had issues with horizontal scrolling of tags vs swiping through suggestions. One participant used a swipe left to navigate to the next image without publishing.
    • 💥We’re going to stack tags (again) instead of outputting them on one line with a horizontal scroll area. Changes will be applied in T247022#5980868.
  • 👎3/5 had issues with publishing of tags (different errors, e.g.: “Save has failed“, “503 Publish Error“, No connection)
    • 💥We probably need to investigate on this. Is the API reliable in areas with slower internet connection?

Participants

Tested with 5 random Android participants from India with the latest Wikipedia Alpha 2.7.50309-alpha-2020-03-17. Check out the “Add image tags on Android“ session usertesting.com.

Legend

👍= Works well

👎= Not working well

❓= Not sure if good or bad

🗣= Participant suggests this

💥= Robin’s assessment

Summary / action items

  • 👍5/5 were zooming the suggested images naturally by pinching without being asked to do so.
  • 👍5/5 mentioned that the onboarding is pretty clear and they were able to correctly rephrase it in their own words.
  • 👍4/5 use both “Add tags“ and computer suggested tags. Interestingly in no particular order, sometimes they tap “Add tag“ first sometimes they go with the computer suggested ones.
    • 💥Positioning of “Add tag“ might be not that important. Users approach it with no particular process in mind (in contrary to our internal view/story). We’re going to stack tags (again) instead of outputting them on one line with a horizontal scroll area. Changes will be applied in T247022#5980868.
  • 👍3/5 used image description at the top to add tags with the “Add tag“ functionality.
    • 💥Image description adds value in most cases, exactly how we intended it. No action item needed.
  • 👎4/5 had issues with horizontal scrolling of tags vs swiping through suggestions. One participant used a swipe left to navigate to the next image without publishing.
    • 💥Disable swipe gestures completely to make sure people can horizontally scroll tags without issues and are not using swipe gestures to navigate to the next image without publishing. Action item!
  • 👎3/5 had issues with publishing of tags (two different errors: “Save has failed“ and “503 Publish Error“
    • 💥We probably need to investigate on this? Is the API reliable in areas with slightly slower connection?
  • ❓2/5 asked for thinking about hashtags (#), as they’re used on social media platforms like Twitter, Instagram or Facebook.
    • 💥Might be worth to include the notion of a hashtag into designs at some point

Usability tests

mrvig

(a crazy fast typer)

👍Zoom’s image as the first action, has no issues with it.

👍Had a good experience, “it’s really easy!”

👍Interface is pretty clear to him.

👍Swipes to next image without an issue.

👍Add’s a custom tag right away.

👎Completely ignores the computer suggested tags at first and thinks they’re already added to the image (doesn’t tap it)

👎Not sure if he has to publish the images.

👎Doesn’t use description at the top.

❓Wants to use #hashtag

❓Thinks that the search behind “Add tag“ lists computer suggested tags.

❓Adds generic tags with the custom tags feature.

🗣Suggests that users should be able to create their own tags (and not choose from a list)

user4875

👍Thinks the onboarding dialog is easy to understand, everything is clear.

👍Has no issues zooming an image.

👍Uses “Add tags“ and computer suggested tags. The first thing he tapped on is “Add tags“.

👎Does not tap on “Publish“ for almost all images. Swipes right after tapping on a tag.

🗣Likes to have custom hashtags.

NoxDRaz

👍Uses the interface perfectly (add tags and suggested tags).

👍Sees computer aided tags first

👍Uses “Add tags“ right away.

👍Uses “Publish“ right away.

👍Uses description at the top to add tags.

👍Pinches to zoom perfectly

👎“Add tag“ search does not work half of the time (connection?)

👎Publishing tags feature doesn’t work (connection?)

👎“This Alpha version is pretty bad“ (connection?)

🗣Feature does not work properly. Internet connection issues? “Save has failed“ error message all the time.

Di4barfi

👍Generally understands the onboarding dialog. Even mentions AI supported image tagging.

👍Intuitively zooms images.

👍Uses computer suggested tags first. Order of CAT / custom does not matter.

👍Seamlessly understands how to add suggested and custom tags.

👍Uses description to add tags.

👎Has issues with horizontal scrolling of tags. Suggests to increase the horizontal scroll height.

❓Sometimes doesn’t find tags (types in too specific terms).

soh78

👍It’s fun adding tags!

👍Understands onboarding dialog.

👍Uses description to add tags.

👍Uses “Add tags“ functionality first.

👍Uses computer aided tags too.

👍Hits Publish and expects to see the next image.

👍Zoom works perfectly.

👎Has some issues with horizontal scroll, suggests to make to horizontal scroll height bigger.

👎Publish error (Code 503), sometimes.

🗣Bigger image.

❓Reads the onboarding dialog and thinks of hashtags (#, like the ones Twitter, Instagram, Facebook)

Event Timeline

@Dbrant - have we updated the current alpha to use the updated/improved API that @Mholloway said was released last week? I'm worried about this:

👎3/5 had issues with publishing of tags (two different errors: “Save has failed“ and “503 Publish Error“
💥We probably need to investigate on this? Is the API reliable in areas with slightly slower connection?

If we're using what will be the production API, then I'm even more worried. We can't send this feature out into the world if a significant number of failures to save are still happening.

cc @dr0ptp4kt

As with the web implementation, tags are published using the wbsetclaim API provided directly by Wikibase, not a custom API. Ramsey said they haven't heard reports of such issues with the web feature when I asked just now. Possibly there's a bug in the app, though I personally haven't run into any trouble publishing tags while testing in the app.

I don't have access to the usertesting.com URL, is is possible to get the recorded interactions somewhere more accessible like a Google Drive?

The 503 error is suggestive of server side error, of course. Would VPN'ing to India help rule out something else funky going on? Anyone know the URLs and the request/response signatures being used by the alpha app?

I might suggest a few people sharing IDEs on video to walk through the code and ensure code is doing what we expect in the different parts of the stack. Let me know if you need any help scheduling.

I was able to reproduce some of these errors by using a VPN to other countries. It looks like these users might have been in IP ranges that are globally blocked, but for some reason we weren't detecting their blocked status in our Suggested Edits screen, so we were allowing them to proceed to editing.
I'll look into why we're not catching the blocked status correctly.

Here’s a link to all sessions on usertesting.com @dr0ptp4kt.

The participants NoxDRaz, soh78 and Shinigami99 experienced issues.


Also some error screenshots (in Spanish) from @Pginer-WMF, who recently trying out the feature (thx!):

Screenshot_20200318-101917.png (1×720 px, 51 KB)
Screenshot_20200318-102054.png (1×720 px, 270 KB)
Screenshot_20200318-102313.png (1×720 px, 628 KB)

The Varnish web request 50x logs in Logstash (which I believe are sampled) are showing occasional 503s for various requests from WikipediaApp/ user-agents, not only wbsetlabel but also wbsetclaim (setting image captions) and echomarkread (marking notifications as read): https://logstash.wikimedia.org/goto/dd02d4649cc045b03727424a11346f94

I found an issue on OkHttp suggesting that web servers may respond with 503s to aggressive retry behavior by OkHttp: https://github.com/square/okhttp/issues/4253

That would line up with our only seeing reports of 503s from people who are also reporting connection failures.

I found an issue on OkHttp suggesting that web servers may respond with 503s to aggressive retry behavior by OkHttp: https://github.com/square/okhttp/issues/4253

This doesn't seem to be quite right; after combing through the web request logs, I don't see any evidence of the Android app aggressively spamming retry requests.

However, there has been a general increase in 503s returned from POST requests to api.php (EDIT: across the board) since March 15 — mostly, but not exclusively, for requests coming through the esams (Amsterdam) and eqsin (Singapore) DCs: https://logstash.wikimedia.org/goto/61ae12fb0cdadeffa92f4d3c86e8f4dd

I'll open a separate ticket about this.

Thanks @Mholloway - if you could link the ticket here that would be great.

Besides that I don't see anything else we need to do on this ticket, unless you do @schoenbaechler ?

@Charlotte

Besides that I don't see anything else we need to do on this ticket, unless you do @schoenbaechler ?

Precisely, the other stuff is handled in T247022