Page MenuHomePhabricator

Setup and run Userlytics test of article summarization
Closed, ResolvedPublic

Description

Background

  • We need to setup and run a test on Userlytics to verify article summarization functionality

User story

  • As a team looking to run user testing, we need to set up and run a test using userlytics to verify our hypothesis with a subset of users

Requirements

  • We have identified audience criteria
  • We have setup the test
  • We have run the test against the identified users
  • We have summarized results

This task was created by Version 1.2.0 of the Web team task template using phabulous

Event Timeline

Jdlrobson triaged this task as Medium priority.Sep 12 2024, 4:26 PM
JScherer-WMF raised the priority of this task from Medium to High.Sep 23 2024, 4:37 PM

Experiment is complete and analysis is written.

tl;dr

We might have a hit on our hands with this feature. Participants found it easy to use, useful, and had an appropriate level of trust in the machine-generated summary. Responses were much more positive than I expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed.

ovasileva subscribed.

Experiment is complete and analysis is written.

tl;dr

We might have a hit on our hands with this feature. Participants found it easy to use, useful, and had an appropriate level of trust in the machine-generated summary. Responses were much more positive than I expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed.

@JScherer-WMF - would it be okay to publish the protocol as well prior to resolving? I'll then move it to our main documentation on mw

Protocol

  1. You are looking at the top section of the Wikipedia article about Dopamine. Without clicking anything, find the section labelled "Simple Summary". Based on what you see, describe in your own words what you think the Simple Summary section does.
  2. Without clicking anything, describe what information, if any, is missing from the Simple Summary section.
  3. Without clicking anything, please describe what you think "Unverified" means.
  4. What do you expect to happen if you click on the Simple Summary section?
  5. Click on the Simple Summary section. Did it do what you expected? Why or why not?
  6. Please describe in your own words what a Simple Summary is.
  7. Please indicate which section or sections of the page were generated by AI. What makes you think an AI created those sections of the page?
  8. What information, if any, is missing from this page?
  9. If you wanted to find out more about the Simple Summary of this article what steps would you take?
  10. Would you trust the information presented in the Simple Summary? Why or why not?
  11. What aspect of the Simple Summary section confused you?
  12. What ideas do you have about how Simple Summaries could be improved?
  13. What do you think about seeing features like Simple Summaries on Wikipedia?