met with @MRaishWMF and discussed approaches (noted below). will meet with Web team next Tuesday to clarify 1) what the team's most prioritized research goal is and 2) what capacity they have to provide the testing artifacts and/or environment to meet the goal. after confirmation, we can begin work.

~~~

1: you want to know more about how the summaries are received, how participants interact and respond to them naturally, how it affects their comprehension and reading experience, and their general sentiments wrt trust, AI, AI on WP, etc (more in-depth but similar to justin's previous testing). no add'l effort needed as we can just use the browser extension, but you will not necessarily get direct information about the effect of bad quality summaries.
2a: you want to know more specifically about negative effects of seeing bad quality summaries. slightly higher effort with static mock-ups or minimally interactive artifacts of test articles in 2-3 topic categories and generated good/bad summaries for each.
2b: you want to know both 1+2a. very high effort with a built out test/beta wiki environment with test articles/summaries.

In consultation with the Web team and Research team colleagues, we are approaching alignment on a structure for this research, as outlined in the research brief. Briefly, we are planning to use panel participants (sourced externally) to investigate what happens when people interact with good or bad summaries in more difficult or easier articles. Next steps will include aligning with the team on experimental materials, further consultation with Research colleagues, and setting up the instruments on Qualtrics as we move toward the pilot stage.

(Moving the project to Backlog until we decide on the question of how many languages to include. As I have learned from Mike, this can affect the overall study scope and design and may need a few more days of iteration. I'm working with Mike on it and will be led by him to involve other folks as needed to finalize the scoping piece.)

Mike and I discussed and decided to keep the scope of this project focused on English. From the now updates Research Brief: "Once this study has concluded, English may serve as a “proof of concept” for testing or evaluating this feature, and the study may be extendable to other non-English languages depending on the Web team’s rollout plan.". We also updated the timelines to leave sufficient room for analyses, internal review in the Design Research team, and shareout.

Justin and Olga as the stakeholders for this projects signed off on the scope and phases/timeline.

I updated the task deadline here. The rest is in Mike's capable hands.

After ongoing consultation with Research team colleagues, this project has moved into the pilot testing phase. @JScherer-WMF has helpfully provided static mockups to be used as visual artifacts, and they are being shown to participants in a survey hosted on Qualtrics. Currently, participants entering the survey have an equal chance of being routed into the "good summary," "bad summary," and "control (no summary" conditions, after which they have an equal chance of seeing article pairs easyA+hardA, hardA+easyA, easyB+hardB, or hardB+easyB. There is a pre-test measuring trust in and knowledge of Wikipedia, AI usage, and selected demographic variables, and a post-test measuring opinions about AI+Wikipedia, followed by a repetition of the Wikipedia trust questions shown in the pre-test.

The survey appears to be functioning as intended, however further consultations are planned to determine if a more complete randomization of articles is appropriate (e.g., easyB+hardB, easyB+hardA, hardA+easyA, hardA+easyB, etc.) is ideal. Also, a small number of the static mockups currently have red underlining indicating misspellings under some of the content (e.g., latin species names) which will ideally be removed before we go "live" with larger numbers of Prolific participants.

A more-or-less complete draft of the final report has been delivered to the Web team, and a larger readout has been scheduled for Monday: WMF-internal link to Simple Summaries report