I've started working on the script to generate the dataset and I've come across a few questions / things to be confirmed:

1. The proposed quality score of > 0.85 removes most candidate articles - including Featured Articles, which often score below the threshold. I did some test runs on mathematics and physics and this filter left too few articles to fill 50 seeds per topic as it often removes 75-90% of candidates that passed remaining filters. I'd suggest lowering the quality bar a little to maybe 0.70?

2. Are we okay with random sample of articles from 70th-90th percentile of pageviews? Some topics contain millions of articles and iterating over them with API would be expensive. I'd suggest we take random sample of e.g. 10000 articles per topic, calculate popularity range in this sample and get candidates form this sample.

3. Is it okay to use last 60 days to measure the popularity of articles? A longer window would be more stable, but requires us to gather data from historical dumps, whereas we can conveniently query last 60 days range via API.

4. Should we also run the quality checks on the articles recommended via MoreLike API? Currently the description reads like we should only run it on seed articles and I'm not sure it's intended as recommendations would be 75% of the dataset. If yes, we'd take next-best MoreLike result whenever ones fails our checks.

5. The keyword filter is implemented, just waiting for the designated keyword list.

Thank you so much Bartosz!!!

One thing I should have written in the original ticket is that I think it's better to filter too conservatively for now, have some questionable results, and have to add more filters later, than to overfilter and not know which levers to pull to get more articles. Answering these questions with that philosophy in mind.

1. Absolutely, we can lower the threshold. I chose 0.85 somewhat arbitrarily, after testing a few articles on this Toolforge interface, and I did not rely on any knowledge of the distribution of quality scores in each of these topics. We can totally make this 0.7 or even 0.6 - and if it makes more sense to use a percentile-based filter (vs. static values) I think it would also be fine to say top ~35% of article quality scores for that topic.

2. Makes a lot of sense, sounds good!

3. Yes totally, let's use a 60 day window. @ABendall-WMF just tagging you here for visibility - if we generate this list now (in July) but run the experiment in October/November, the pageview stats we're using will be a little old. I personally think that's ok because we're not trying to optimize for content trendiness, but please let us know if you want us to change this approach.

4. Great question! The Apps team hypothesizes that the recommended articles from MoreLike will be generally good enough to use, if the seed article is good enough. Maybe we can start with a list of what the unfiltered recommendations would be per seed, to see if this hypothesis checks out? And if it doesn't we can add those filters later?

5. Sorry for not including this earlier. Let's start by using this keyword list: https://github.com/censor-text/profanity-list/blob/main/list/en.txt

Also, in our sync today we talked about adding a volatility filter to weed out articles that are being edited too often. Can we please do two things here?

1. Add a filter for Edit rate (# edits/day). Assuming this experiment runs for 30 days, the goal is to get a list of articles that are less likely to be edited during that time. There are two types of volatility: continuous volatility (this article always gets edited often) and sudden volatility (something has happened that is making this article get edited a lot now). I'm gathering some information in this slack thread so that we can construct a heuristic to start with. Note: This is another filter whose impact might be limited by us building this list in July for an October experiment, but let's see what we get for now. Perhaps we focus on getting the query right for now, but actually define the list and generate the files closer to launch.
2. Add a column for Edit rate (# edits/day for last 7 days) as a column in the spreadsheet

Ok based on the discussion in the Slack thread I think we should use "edited in the last 24 hours" as a proxy for editing volatility. I'll update the ticket details at the top to capture this, but please let me know if you disagree with this approach or have any questions - totally happy to try it a different way!

I've performed first full run of dataset generation and I'm attaching the resulting dataset csv file here (columns: Topic, Article title, Article URL, Type, Edit rate in the last 7 days).

Note that the pipeline is fully repeatable so we can easily regenerate the final list closer to the October launch.

To confirm how the seeds and recommendations were selected:

• Candidate pool - all Featured Articles in the topic + a random sample of ~10000 articles out of which ~2000 fall in the 70-90th percentile of pageviews in the last 60 days.
• Filters in order: at least 10 langlinks, not edited in the last 24 hours, no words from the keyword list in the lead section, article quality score >= 0.70
• Remaining candidates sorted by topic score and top 50 taken as seeds
• Recommendations were gathered using top 3 MoreLike results for each seed, no filters run there. If any of the MoreLike results showed up somewhere else in the dataset, I replaced it with next-best MoreLike result so that we don't have duplicates.

With this process we got 6473 articles, 1624 seeds and 4849 recommendations, a little short of our target. Some notes on those results:

1. 14/39 topics did not fill 50 seeds, those were: education (3), mathematics (10), fashion (11), business-and-economics (14), south-america (19), central-america (21), society (25), books (29), africa (38), comics-and-anime/performing-arts/food-and-drink (39), chemistry (40), architecture (47). There seems to be two main causes of this:
   • For education, books, architecture, business: the langlinks >= 10 filter often removed >90% of candidates. Those articles were often locally notable (buildings, companies, schools) and exist in few languages.
   • For mathematics, fashion, chemistry, regional topics: the quality >= 0.70 filter also often removed >90% of the candidates (vs ~50% for topics that filled).
2. One visible effect of unfiltered recommendations - around 150 lists ("List of School Rumble episodes", year pages) made it to the dataset, mostly in comics-and-anime, films and video-games, whereas we didn't have any such pages for seeds. Those might not be the best for TTS experience.
3. 23 recommended articles are missing, where a seed's MoreLike results were already present somewhere else in the dataset.

To mitigate points 1) and 3), we can:

• For topics where we have issues gathering seeds - lower the quality bar (or switch to per-topic percentile) and/or langlinks bar, give wider pageview band, or just accept fewer seeds there? Happy to try any combination of those.
• Gather more MoreLike recommendations to make sure we fill all recommended articles.