Here's a quick prototype showing the interactions

Challenges
If we directly adapt our current component, several issues arise:

Spoke to @jwang. We will start analysis towards the end of this sprint.

ticket for analysis created.

cc @ovasileva

I've taken this to a natural stopping point. Here's a video of the demo.

Current prompting:

I have early access to GitHub Spark for this, so I'm going to use that as a prototyping tool.

@jwang is the above enough for the data platform folks to work with?

For the designs here, I think we should keep it simple to A/B test these things.

I drafted a high-level interim report. More interviews, detailed analysis, and reports to come.

I have a few options for folks to consider.

I'm finished the affinity mapping. I just need to write up the report.

@jwang @ovasileva Do we have an analysis ticket for this yet?

It seems like we've decided on a approach here, so I'm going to move this for sign off and create child tickets.

Looks good!

I'm thinking about MinT and how they were able to create a machine-in-the-loop process for translation. A human editor manually commits and has direct control over every word that ends up on wiki. We could pre-generate summaries, admins could choose which articles become summary candidates, and then we could rely on editors to manually commit the summaries. After a certain number of changes in the source text, we could remove a summary and add it back to the queue.

All good questions! Thanks, Jon.

Here is a WIP list of potential affordances we could consider building:

• Admins choose which articles get summaries and which don't via a community configuration
• A quick way to manually remove summaries from specific articles
• Editors can manually edit summary content like normal article content
• Automatically removing the summary from an article after 5 readers have marked it "unhelpful"
• Automatically removing the summary from an article after more than 20% of readers who have seen it mark it "unhelpful"
• Readers generate a new summary "on demand" every time they open a summary. Editors have no ability to view, moderate, or edit individual summaries
• A community-maintained simple summaries template that editors can add to or remove from any article
• Admins can adjust the prompt engineering for summaries on an entire wiki
• WMF generates the summaries for an entire wiki up-front and automatically publishes them in an "unverified" state
• A reader preference for hiding and showing summaries
• An LLM-powered dashboard that shows the quality of summaries on a wiki, as well as other pertinent metrics
• Automatically re-generate the summary of an article when the article has had XX bytes changed and/or XX number of diffs/edits
• A sandbox to test out the impact of prompts on simple summaries
• Tools to discover the articles on any wiki that would benefit the most from simple summaries
• A list of all summaries currently live on any given wiki

Jan and I discussed this this afternoon. To my mind, the main thing we're looking to learn from this exercise is an answer to the following questions:

@JScherer-WMF @Jdrewniak to pair on this today/tomorrow.

Thanks for the discussion here @matmarex and @MikhailRyazanov.

In T386497#10584477, @SToyofuku-WMF wrote:

Can we also note somewhere what the plan was for if a user dismisses the popup, but then subsequently wants to enable the experiment?

In T386497#10580296, @JScherer-WMF wrote:

In T386497#10579749, @JScherer-WMF wrote:

Some things to do here:

In T386486#10581159, @Jdlrobson-WMF wrote:

3. Wiring up the "Was this summary helpful?" buttons so they do something. (@JScherer-WMF I assume the behaviour here will be identical to browser extension?)

In T386486#10580223, @Jdlrobson-WMF wrote:

@Jdlrobson - is the penalty hit for every page or just pages where there is both a hatnote and summary?

The penalty hit would be for every page with summaries in the test group as this would require us to client side render the summary notice.

I would also lean towards [1]. We can re-evaluate during data QA. Does that make sense?

In T386497#10579749, @JScherer-WMF wrote:

Some things to do here: