What are we testing?

Senior is curious whether the edit they want to make has already been discussed ⟶ senior visits busy article talk page

Junior wants to check in with others before making an edit⟶ junior visits busy article talk page

This script tests for:

• Scanability of TOC
• Can they interpret the metadata in the TOC?
• Can they search for and find an article using the TOC?
• Usability of a maximally used page
• Usability of sticky header

Who is being tested?

Junior and Senior Contributors

What page are we testing

a busy Article Talk page

Senior Tester Findings

Average amount of time to correctly recognize that this page is some sort of message board is 14.8 seconds for desktop and 9.8 seconds for mobile testers.
"It looks like a regular Wikipedia page but I have no idea what it is. These people look like they are commenting on Wikipedia pages." D UT-5

Can they interpret the metadata in the TOC?
✅ 5 of 5 desktop users were able to interpret the metadata
✅ 4 of 5 mobile users were able to interpret the metadata
While testers who discovered the freshness feed found it very effective, there are some discoverability issues , particularly the size.

Can they search for and find an article using the TOC?
✅ 5 of 5 desktop users were able to find an article using the TOC
✅ 5 of 5 mobile users were able to find an article (not using the TOC but using the metadata)
While mobile testers were able to perform this task, it was a big struggle. They searched for the metadata - I would recommend that the metadata be condensed so that it's more likely to live on one line

Usability of a maximally used page

• Testers on desktop were able to use the table of contents to seamlessly navigate the page
• The scrollbar was not displaying on all of the TOC
• The experience continues to be challenging on mobile.
• There is a tension between needing to have a conversation open by default in mobile and also seeing/discovering all of the other topics on the page
• On many of the mobile tests the "Add topic" button would not display

Usability of sticky header

• Testers were easily able to find and use the "add topic" button with the sticky header.
• One tester wanted to confirm that their comment was posted after it happened automatically so they (incorrectly) used the search bar at the top of the page hoping to do an in-page search

Mobile

• There were some quirks to the interface in mobile (the add topic button wasn't showing, it was an old interface etc.)
• I think that these mobile tests should be re-run considering the testing environment
• Several testers commented about the slowness of the prototype, particularly on mobile
• One participant expanded the conversation but didn't notice any change in the interface ( possibly due to there being no obvious visual changes)

Feature Requests:

• Dark mode
• Upvoting
• Topic containers

Requests topics have "boxes" so that he can "easily distinguish what I'm looking at and when the first comment end and the second comment starts." - D UT-2
"Maybe you could downvote negative discussions and they can be hidden after a certain number of downvotes so they don’t clutter up the page and make it harder to find relevant discussions" - D UT-6
"it would have been better if the most recent (conversation) was at the top of the page." D UT-7

Timeline
Most testers assumed (either by scrolling or verbally declaring intent) that the newest conversations would be posted at the top of the timeline (rather than the bottom)

Junior Tester Findings

Average amount of time to correctly recognize that this page is some sort of message board is 11.6 seconds for desktop and 12.8 seconds for mobile testers.
"It looks like a regular Wikipedia page but I have no idea what it is. These people look like they are commenting on Wikipedia pages." D UT-5

Can they interpret the metadata in the TOC?
✅ 5 of 5 desktop users were able to interpret the metadata
✅ 5 of 5 mobile users were able to interpret the metadata
While testers who discovered the freshness feed found it very effective, there are some discoverability issues , particularly the size.

Can they search for and find an article using the TOC?
✅ 2 of 5 desktop users were able to find an article using the TOC
❌ 3 of 5 desktop users attempted to find the article using the sticky header
❌ 3 of 5 mobile users attempted to find the article using the mobile search bar
One mobile tester (UT6) completely got taken off course after missing the reply button and accidentally clicking on a user talk page that was in comment signature.

Timeline
Almost all testers expected the comments to be ordered chronologically with the newest comments at the top of the page.
One tester was confused that some comments in the timeline had the full date (with year) and others did not. (UT-9)
"Personally I would want this to be sorted in chronological order" UT-10

TOC
! the word beginning in the table of contents should be styled differently
Couldn't tell that the TOC was longer than the truncated display (needed some kind of indicator)
One testers (UT-4) gets confused which comment the metadata in the TOC refers to

General Navigation Issues
""Navigating through the website was extremely hard. Even for me, I am proficient at computers and if I can't get through it my grandmother would just throw it through the window." UT-6

Edit
"I see you can edit. Which is odd to me because i would think you would only be able to edit a comment if you were the one who wrote it." UT-10