Page MenuHomePhabricator

Conduct a control test of as-is new discussion workflow
Closed, ResolvedPublic

Description

To make it easier and more intuitive for Junior and Senior Contributors to *participate* on talk pages, the Editing-team is exploring ways to improve the workflow(s) for starting a new discussion.

In an effort to better understand Junior Contributors' current experiences starting new discussions on talk pages, we will conduct a series of user tests.

The results of these tests will inform the design of v1.0 (T243248) and serve as helpful baselines to compare against the findings in T243249.

Test goals

  • To better understand the challenges Junior Contributors' face in the process of starting a new discussion on an article talk page.
  • To better understand the challenges Junior Contributors' face in the process of starting a conversation with on another contributor on their user talk page.
  • To identify how the workflows for starting new discussions on article and user talk pages could be improved.

Research questions

A draft list; design to complete in the process of drafting the test script
For both mobile and desktop, we would like to know:

  • Is the workflow discoverable?
  • What parts of workflow do Junior Contributors have difficulty with?
  • What mistakes do Junior Contributors consistently make in the process of attempting to start new discussions?
  • What about the existing workflow do contributors find to be intuitive?

Workflows to test

A draft list; design and product to complete in the process of drafting the test script

  • Starting a new discussion on an article talk page and on another contributor's talk page.
  • Starting a new topic within an existing conversation (read: adding a 0th level comment in an existing section)

"Done"

  • Finalize testing goals and research questions
  • Write user tests
  • Run test(s)
  • Summarize test findings
  • Create Phab tickets to address test findings

Details

Due Date
Feb 21 2020, 5:00 AM

Event Timeline

ppelberg updated the task description. (Show Details)Jan 22 2020, 2:44 AM

Might it be useful to test on a Project-namespace discussion page as well, such as w:en:Wikipedia:Help desk?

ppelberg reassigned this task from ppelberg to iamjessklein.Jan 22 2020, 4:08 PM
ppelberg added a project: Editing Design.
ppelberg updated the task description. (Show Details)

Might it be useful to test on a Project-namespace discussion page as well, such as w:en:Wikipedia:Help desk?

This is a good idea, @Jc86035 – thank you for raising this. We will consider including Project-namespace discussion pages in this test.

JTannerWMF moved this task from Backlog to New Discussion 1.0 on the OWC2020 board.Feb 5 2020, 2:38 AM
JTannerWMF edited projects, added OWC2020 (New Discussion 1.0); removed OWC2020.
JTannerWMF moved this task from Incoming to Design on the Editing-team board.
ppelberg updated the task description. (Show Details)Feb 10 2020, 1:25 AM
JTannerWMF set Due Date to Feb 21 2020, 5:00 AM.Feb 11 2020, 5:26 PM

Thanks for the input @Jc86035
@ppelberg and @Whatamidoing-WMF Please provide your feedback on the draft protocol.

@ppelberg and @Whatamidoing-WMF Please provide your feedback on the draft protocol.

You got it. Comments are in the doc.

iamjessklein updated the task description. (Show Details)Feb 18 2020, 2:02 PM

Thanks @ppelberg - I've updated the testing protocol so that we will be testing this on usertesting.com mobile and desktop web.
I have the test written and ready to go on usertesting.com and have one question. I'm noticing on the prototype server that the infobox templates aren't visible. Is this something that we can fix before putting the test up? The reason that I am asking for this is so that we can recreate the actual experience as much as possible. cc/ @Esanders @matmarex

English Wikipedia:

Prototype Server:

Update: I ran the usability tests without the templates at the top of the page. I did five tests on mobile and five tests on desktop (both web). The next step here is reviewing and synthesizing the tests.

iamjessklein updated the task description. (Show Details)Feb 24 2020, 1:55 PM

Findings:

We ran a test on usertesting.com on February 21, 2019 - The test recruited 10 random, technically - advanced web users. Participants were directed to an article page set up on the prototype server . The detailed findings can be found on limited access test log.

  • 10 tests were conducted
  • Participation: Desktop: 1 male ; 4 female// Mobile : 2 male/ 3 female
  • 5 participants were desktop web users
  • 5 participants were mobile web users
  • All participants had familiarity with Wikipedia and some participants had previously edited an article.

Identifying where to add a new discussion
✅4/5 mobile web participants successfully identified the button to add a new discussion
✅5/5 desktop web participants successfully identified the tab to add a new discussion

Adding a new discussion
✅2/5 mobile web participants successfully added a new discussion to the Talk page
🚫1/5 mobile web participants edited the talk page but did not add a new discussion
🚫1/5 mobile web participants unsuccessfully completed task
🚫1/5 mobile web participants gave up on task
✅3/5 desktop web participants successfully added a new discussion to the Talk page
🚫1/5 desktop web participants unsuccessfully completed task
✏️ None of those who posted were able to sign their comments correctly.
✏️ The participants who unsuccessfully completed the task edited the page and added a comment but not a new discussion
✏️ Many participants were uncertain that their actions would lead them to a successful edit.
“yes it was very easy to use and straightforward. i found what i needed immediate and was able to add a discussion”- CT-D5
“i really don't know if i'm doing the right thing" - CT-M5
“It was very confusing how to edit this page. I’m not sure if i was taken to the wrong page or if i needed to know something extra to successfully edit this page." - CT-M1

Locating discussion in the Talk page
✅1/5 mobile web participants successfully located their discussion in the Talk page
🚫3/5 mobile web participants unsuccessfully completed task
🚫1/5 mobile web participants gave up prior to completing task
✅4/5 desktop web participants successfully located their discussion in the Talk page
🚫1/5 desktop web participants unsuccessfully completed task
✏️ For many participants, hunting and not finding their post signaled to them that they didn’t actually complete the task as they had intended.
"i don't see where I published the topic or if I am doing it wrong."" CT-M3
“I like that as soon as I published it went straight to the section and was easy to see.”- CT-D3
"Absolutely, no trouble whatsoever. All the tools you need to add, edit information are right there for you to see. No searching or looking all over the page to find it." CT-D2

Miscellaneous Notes
✏️Many participants did not realize that they were viewing a Talk page and provided comments as if they were looking at an Article page
✏️Several participants were put off by having to provide comments in Wikitext. One participant said that they expected to see more of a “forum.”
✏️This test validated some general ux changes that are already being integrated into features. This includes: automating summary, automating signatures, adding more visual editing.
✏️Several participants got lost in the course of their test and had to find the Talk page again through using the search bar.
“it would be nice if when you click "show preview" it looks exactly what it would look like on the page.” CT-D3
"I would say the only thing I was confused about what the summary box when you click "Edit Source" as I did not seem clear what was the purpose of it, as writing in the text box seemed to introduce the text anyways".
"i know that i can trust this tool." CT-M2
"i find it a little difficult to navigate. i wasn't sure if i needed to know how to code .... obviously i don't because I messed up." CT-M1

Recommendations:

  1. To Test:
  • Test on different namespaces.
  • Prepare a test with “top” templates - I suspect it was easy for desktop web participants to identify the navigation and buttons with such ease because this was removed from the page (and therefore this isn’t a typical user scenario)
  • Test with logged in participants.
  • Test separately for Advanced Mobile Contribution interfaces
  1. To Workflow:
  • Improve discoverability of the “Add a new discussion” button. Although desktop web users were able to successfully identify the button, many did not truly understand what they were doing until it was published.
  • Improve workflow for writing an initial comment on a thread. Many participants only left a topic and were not directed how to write the initial comment.
  • Improve discoverability after posting new discussion.
  • Differentiate between the design of the article and talk pages. This was pointed out in the Talk page consultation already, but definitely validated through testing.
  • Consider having the visual editing reply optimized in the UI hierarchy so that junior contributors aren’t turned off from posting due to unfamiliar characters.
iamjessklein updated the task description. (Show Details)Mar 2 2020, 5:13 PM
iamjessklein moved this task from 📐 Doing to 👀 Needs PM Input on the Editing Design board.
iamjessklein added a subscriber: iamjessklein.
ppelberg closed this task as Resolved.Mar 12 2020, 2:06 AM

The above summary is helpful, Jess – thank you for pulling this all together.

I've used the findings you shared in T243251#5933544 to inform the requirements in this task: T234825#5962717.


Note, as you mentioned above, and we've talked about off-Phabricator, there are some additional things we would like to learn from another round of user testing. Those "things" include:

  • How do contributors find the "starting a new discussion workflow" on user talk pages?
  • How do contributors find the "starting a new discussion workflow" on article talk pages when templates like {{Talk header}} are present?
  • How do contributors go about cancelling the conversations they started to create?
  • How do contributors go about formatting the content of their discussion using wikitext?
  • How do contributors expect to find out when someone comments on/in the discussion they start?

I've represented the above in the following task: T247485

Great. @Esanders and I just made new test page - that has working templates!!! I'll update the test progress on this ticket.

Great. @Esanders and I just made new test page - that has working templates!!! I'll update the test progress on this ticket.

Oh, this looks great. Awesome, Ed and Jess.