This spike is time bound to 2 days and will report out and sync on whether more time is needed, where the gaps are etc.

@Dbrant I had the wrong patrol link. I think this would be the more accurate one to look at: https://en.wikipedia.org/wiki/Wikipedia:New_pages_patrol

After conversations with Sarah & Samuel, I am updating the priority of understanding if we can filter by patrolled pages, or pages with more than 1 unique editor as a proxy. Deprioritizing the 90 day waiting period, as that would cause more stale suggestions.

Submitted for your approval:

Preamble

The one major prerequisite for getting new articles by Topic (or any new Feed content type, generally) is that it must be a single API call that produces a list of candidate articles for us to present. Anything that requires additional API call(s) per article would be a non-starter, for reasons of performance, latency, and rate limiting.

The basics

CirrusSearch makes it possible to get a list of articles ✅, within a certain Topic ✅, sorted by creation date ✅, which gets us what we want in a very basic way:

https://en.wikipedia.org/w/api.php?action=query&generator=search&gsrsearch=articletopic%3Abiography&gsrnamespace=0&gsrsort=create_timestamp_desc

The key is to include the parameter gsrsort=create_timestamp_desc, which sorts the results by descending timestamp, effectively giving the "newest" articles within this topic.
This fulfills the fundamental requirement of: "Do we have a scalable way to present new articles by topic within the feed?"

However, this does not give us the actual creation date of each article; it just sorts them by creation date. To get the creation date of each article requires additional API calls per article, which is a non-starter. This rules out things like:

• Displaying the creation date of the article in the feed.
• Limiting our display of new articles to only those with a creation date of more than 90 days ago.

Non-English caveats: Non-English language wikis will probably be wildly variable in terms of the quantity, quality, and freshness of "new" articles.

Patrolling

The whole concept of “patrolling new pages” was designed and built for English Wikipedia, and has not been rolled out on any other language wikis, which likely makes this a non-starter.

If we want to proceed with English only:
To get the patrolled ("reviewed") status of an article is possible, inline with our single query:

https://en.wikipedia.org/w/api.php?action=query&prop=isreviewed&generator=search&gsrsearch=articletopic%3Abiography&gsrnamespace=0&gsrsort=create_timestamp_desc

However, whether each article is reviewed is only exposed as a returned property, not as a searchable field in CirrusSearch. This means that the above query will return isreviewed as true or false for each article, and we'll need to filter out only the true ones. But this also means that we can’t "guarantee" that this single query will always return four (or however many we want) articles that are definitely reviewed. It could be more than four, or it could be none.

Deletion/notability

Determining whether an article is "nominated for deletion" is not a straightforward question.

In principle, this type of question could be answered by searching whether the article contains a specific template. (Nominating for deletion just means adding a special template into the article, nothing more.)

Searching for a specific template (or absence of specific template) can be inlined with our CirrusSearch query:

https://en.wikipedia.org/w/api.php?action=query&generator=search&gsrsearch=articletopic%3Abiography%20-hastemplate%3Anotability&gsrnamespace=0&gsrsort=create_timestamp_desc

Notice the hastemplate parameter (or -hastemplate for negation) that searches for a specific template.

But then the question becomes: which template do we search for? Taking only speedy deletion into consideration, there are at least twenty possible templates to look for, each of which represents a different reason for nominating for deletion. This is all in addition to the "main" templates of Template:Article for deletion and Template:Article for deletion/dated.

Do we also want to look for templates relating to notability issues (Template:Notability)? candidates for draftification? Things like Template:POV, Template:Expert_needed, etc?

Do we want to support languages other than English? Every other language wiki has its own templates to indicate deletion or notability issues (not to mention their own policies around when and how to apply such templates). We would need to research all of these templates, for every language.

Miscellaneous

Can we surface new articles related to users' article interests in any way?

Not really. Technically it is possible to combine the morelike API with the same sorting parameter that we used above (sorting by descending creation time), but this produces results that are as good as random (the morelike signal vanishes).

For how many of the Topics can we surface 4 new articles every day consistently?

This cannot be known without a longitudinal study where we track new articles for every topic, for every language wiki, over the course of X days.

Could we show new articles coming in in (relatively) real time for a given topic?

I really don't think the frequency of new articles is high enough to merit something like this.
As an initial data point, over the course of the last 24 hours at the time of this writing (on enwiki), there have been fifteen (15) new articles under the biology topic, and twelve (12) new articles under the visual-arts topic, for example. This would make it very wasteful to poll the API for new articles at intervals shorter than an ~hour, much less real-time. (And "polling" is all we can do; there's no way for the server to "push" to us whenever a new article-in-a-topic is created)

What information can we access about the new articles that can be leveraged in the design or for filtering on quality? (i.e. patrol status, number of unique editors, size, date created, number of recent edits, presence of image)

Of the things listed in the question, only "patrol status" (discussed earlier) and "presence of image" can be rolled into our query. The others would require additional API calls per article, which is a non-starter.

CirrusSearch makes it possible to get a list of articles ✅, within a certain Topic ✅, sorted by creation date ✅, which gets us what we want in a very basic way:

Amazing woot!

The whole concept of “patrolling new pages” was designed and built for English Wikipedia, and has not been rolled out on any other language wikis, which likely makes this a non-starter.

Even if we went out to all languages, I think it'd still be worth including this specifically for English. Is it at all possible to filter to those that been edited by more than 1 unique editor? (It's okay if you haven't explored this yet. But this is what I see Weeklypedia doing, so I'm wondering if maybe they're doing it by hand.)

This rules out things like limiting our display of new articles to only those with a creation date of more than 90 days ago.

Trying to think of other creative ways to proxy a little bit of a time delay if we can't filter by date created. Instead of adding a date delay, could we do something like instead of the 1st article available under Biology, we would take the 10th most recent something? Do we always only get 10 articles, or could we get more in a single call?

In T431437#12107110, @HNordeenWMF wrote:

Even if we went out to all languages, I think it'd still be worth including this specifically for English. Is it at all possible to filter to those that been edited by more than 1 unique editor? (It's okay if you haven't explored this yet. But this is what I see Weeklypedia doing, so I'm wondering if maybe they're doing it by hand.)

This would require making API calls for each article, which we cannot do. Weeklypedia does't do anything by hand, but it does all of its processing once (1) per week, then sends the results to their subscribers; We would need to do this processing 6 million times per day, once per app install. The way to work around this is to build a different API, or to build and maintain new dedicated web services for this specific purpose.

Trying to think of other creative ways to proxy a little bit of a time delay if we can't filter by date created. Instead of adding a date delay, could we do something like instead of the 1st article available under Biology, we would take the 10th most recent something? Do we always only get 10 articles, or could we get more in a single call?

We can request up to 50 articles at once, and discard the first 10, if we like.

Ok thanks! This is enough to go on for now. I will sync with Samuel on the community implications of filtering like this.

Looking at the equivalent of the New Pages Patrol in other languages, it really seems to vary a lot. For instance, Spanish Wikipedia appears to have a "Librarian" or "Verifier" role that marks pages as reviewed.

Regarding filtering out pages nominated for deletion, the Community Configuration settings for Growth team features offer an example of one approach we could take. You can also see in those settings the two categories en-wiki has currently decided to exclude: "Articles for deletion" and "Candidates for speedy deletion" (which I believe includes articles tagged with any of the speedy deletion templates).

From Johan: At Swedish Wikipedia and many medium-sized wikis, the patrol process for new articles looks like this.