(Still being estimated - @Jdrewniak since there was confusion in the estimation thread could you make sure this gets covered in our next refinement meeting synchronously?)

Estimations are in but are pretty varied which suggests this needs an in sync discussion around scope before being committed to:

• 5 * 2
• 3 x 3
• 2 * 2

After discussing in standup today, we have pulled this out of the sprint board to make space for T387400 when estimated.

@JScherer-WMF @Jdrewniak to pair on this today/tomorrow.

Jan and I discussed this this afternoon. To my mind, the main thing we're looking to learn from this exercise is an answer to the following questions:

1. What affordances, if any, might contributors want/need in order to successfully manage simple summaries on the wikis?
2. How might contributors "verify" simple summaries on the wikis?

To the first question: we have a list of potential affordances that we could make available for communities. What we need to move forward is a list of which affordances contributors think will be the most/least useful. Once we have an understanding of which affordances seem the most useful, we can create prototypes of those affordances and test how useful they actually are.

This lead me to ask the question: What would a contributor need to see and understand before giving us an informed opinion about the affordances that we might build. I think that the background information on the project page, especially the potential reader benefit these summaries could create, and some video walkthroughs of randomly selected articles on desktop and mobile would be enough as a pre-amble before a survey. Perhaps we could provide an optional link to the summaries browser extension on the project page if folks want to dig into the reading experience a bit further.

If that's true, we could run this as a survey like we did for the API survey we ran last quarter. It could look something like this:

Except for the quicksurvey configuration, we could keep this initial stage of the hypothesis to a low/no code level and use Google Forms. using MediaWiki like we did with the typography work last year is also an option and would have its own set of tradeoffs. We would need to collaborate with folks in research to design the survey.

@ovasileva @sgrabarczuk what do you think of this approach?

Here is a WIP list of potential affordances we could consider building:

• Admins choose which articles get summaries and which don't via a community configuration
• A quick way to manually remove summaries from specific articles
• Editors can manually edit summary content like normal article content
• Automatically removing the summary from an article after 5 readers have marked it "unhelpful"
• Automatically removing the summary from an article after more than 20% of readers who have seen it mark it "unhelpful"
• Readers generate a new summary "on demand" every time they open a summary. Editors have no ability to view, moderate, or edit individual summaries
• A community-maintained simple summaries template that editors can add to or remove from any article
• Admins can adjust the prompt engineering for summaries on an entire wiki
• WMF generates the summaries for an entire wiki up-front and automatically publishes them in an "unverified" state
• A reader preference for hiding and showing summaries
• An LLM-powered dashboard that shows the quality of summaries on a wiki, as well as other pertinent metrics
• Automatically re-generate the summary of an article when the article has had XX bytes changed and/or XX number of diffs/edits
• A sandbox to test out the impact of prompts on simple summaries
• Tools to discover the articles on any wiki that would benefit the most from simple summaries
• A list of all summaries currently live on any given wiki

This is an early draft, and I'll seek out input from others in the foundation to add/remove items.

It's worthwhile considering that, in our efforts to test these out with an audience of ~3000 readers, we have used the following affordances:

• WMF generates the summaries for an entire wiki up-front and automatically publishes them in an "unverified" state
• Curating a central list of which articles get summaries and which don't
• APIs and other programmatic tools to ascertain...
  • Which articles are complicated enough to benefit from summaries
  • Which articles are popular enough to have many readers benefit from summaries
  • Which articles are high enough quality to ensure that the LLM has good enough source material
  • Scoring for how simple summaries are after they've been generated compared to the source material
• A list of all summaries currently live on any given wiki (spot checking)
• A sandbox to test out the impact of prompts on simple summaries
• Basic analytics to see how many readers find summaries (un)helpful

Some random thoughts as I read through this (sorry in advance if this is premature in the research process!)

Admins choose which articles get summaries and which don't via a community configuration

What would this look like? Categories? Specific article titles? A template? If pages get moved (think redirects) does the config update?

Editors can manually edit summary content like normal article content

Do we need to put any kind of assistance in place. For example what if a summary becomes too technical and hard to read. Is this something we want to guide editors better with? e.g. AI writing assistance? (grammarly does a good job of this for example for reference - although we can't use that here)

Automatically removing the summary

What is a reader here? Anon? IP masked? Could this be used as a source of vandalism/harrassment?

All good questions! Thanks, Jon.

In T385501#10624862, @Jdlrobson-WMF wrote:

Admins choose which articles get summaries and which don't via a community configuration

What would this look like? Categories? Specific article titles? A template? If pages get moved (think redirects) does the config update?

I don't know. So far, we've used specific articles, and category filters, as far as I know. Good point about pages moving. That'll have to be a requirement to consider in the future.

Editors can manually edit summary content like normal article content

Do we need to put any kind of assistance in place. For example what if a summary becomes too technical and hard to read. Is this something we want to guide editors better with? e.g. AI writing assistance? (grammarly does a good job of this for example for reference - although we can't use that here)

Great question. I was thinking one possible feedback loop could be a combination of manual editing and the helpful/not helpful feedback buttons. If a summary is edited to the point where it is no longer simple, I assume readers would find it unhelpful. If they find it unhelpful, I assume they would downvote it. If we automatically remove the summary after a certain number of downvotes, it could automatically get put in a queue for automatic regeneration, essentially wiping the slate clean, at which point editors could edit again, and so on.... There are a lot of assumptions in there, though.

Automatically removing the summary

What is a reader here? Anon? IP masked?

I assumed all readers who have summaries turned on. I don't know how we would roll this out to different subsets of readers.

Could this be used as a source of vandalism/harrassment?

Great question. I'm not sure. We should dig into this.

I'm thinking about MinT and how they were able to create a machine-in-the-loop process for translation. A human editor manually commits and has direct control over every word that ends up on wiki. We could pre-generate summaries, admins could choose which articles become summary candidates, and then we could rely on editors to manually commit the summaries. After a certain number of changes in the source text, we could remove a summary and add it back to the queue.

It might look something like this:

• Contributors would have much more control over the summaries that make it on wiki.
• This would de-risk many aspects of the summary content.
• This approach is much more aligned with the Foundation's strategy on machine content.
• Summaries would be a lot more rare because they would need to be manually applied, and fewer readers would benefit from them.
• Wikis with less active contributor communities would have fewer summaries, and their readers would benefit even less.

It seems like we've decided on a approach here, so I'm going to move this for sign off and create child tickets.

Next steps after meeting:

• Proceed with proposed design
• @ovasileva Create another ticket to:
  • Set up survey
  • Limit number of options discovered