Some early notes:

API Options

1. We run this API call, and reference the weighted_tags key to get a list of topics. The benefit of this call is that we can pass in multiple titles (up to 50):

https://en.wikipedia.org/w/api.php?format=json&formatversion=2&errorformat=html&errorsuselocal=1&action=query&prop=cirrusdoc&titles=Cat|Dog

2. We run this API call (liftwing article topic API call determined from https://wikitech.wikimedia.org/wiki/ORES#Guide_to_migrate_from_ORES_to_Lift_Wing)

curl https://api.wikimedia.org/service/lw/inference/v1/models/enwiki-articletopic:predict -X POST -d '{"rev_id": 1257586282}'

I passed a "Cat" article revision ID. It returns a list of all topics and their weighting.

Approach Options

1. We have a persisted list of all articles the user read for the last year. The original thinking was to chunk up this list into batches of 50, and make an API call to the action API to get topics for each batch of 50 articles, and persist those topics to device. Due to the nature of the feature, where we preprocess the year in review report, these API calls would need to be made for all users upon first launch. Once all topics are persisted and we determine their final top 3 articles, then the year in review entry point would be revealed. The concern here is that this would be a rather heavy load initially on the servers and for each device for all users that launch the app.

2. Alternatively, we can fetch article topics at the same time as we fetch the article html as the user reads an article, and persist those topics. This way, the topic API calls are spread throughout the year. Then when it's time to generate a year-in-review report, we can skip the API calls and reference the already-persisted list of topics. There are a couple of dashboards that may be helpful in determining an expected requests per second here:
   
   A. This pageviews dashboard in Turnillo. These are pageviews-per-hour, for the last 7 days, from the apps.
   
   B. This grafana dashboard for the mobile-html endpoint. Be sure to select the --domain_v1_page_mobile-html endpoint at the bottom.
   
   Note that the results from each are pretty different. My guess is that the grafana results are the most accurate of what to expect here regarding server load to these topics APIs. Also note that each of these dashboards are for both apps, not just iOS.

@Jelto, @calbon, @Jgiannelos

The iOS team is considering the best way to tackle a topics slide for a future iteration of our year in review feature. It would look something like this:

My proposal is for the iOS app to fetch article topics whenever a user reads an article and persist those topics to device. At the end of the year we will have everything we need for powering a screen like this, local to the device. Do you think an approach like this would cause server load issues? I did some digging on expected requests per second, and I think the most accurate dashboard for this would be this granfana link:

https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?orgId=1&viewPanel=12&from=now-7d&to=now&var-dc=thanos&var-site=codfw&var-service=mobileapps&var-prometheus=k8s&var-container_name=All

If you filter the endpoint by --domain_v1_page_mobile-html_--title_--revision-_--tid, which is our article html endpoint when viewing an article, we get this. The maximum is about 1.40K requests per second from the past 7 days. Note that this amount includes both iOS and Android apps, so I expect it to be lower as long as only iOS fetches these topics.

Tagging folks that might best be able to answer this or steer me right. Feel free to tag others if I'm missing someone. Thanks!

In T379119#10334120, @Tsevener wrote:

@Jelto, @calbon, @Jgiannelos

Sorry for the late reply, I totally missed the tag and thanks for the ping in slack!

My proposal is for the iOS app to fetch article topics whenever a user reads an article and persist those topics to device. At the end of the year we will have everything we need for powering a screen like this, local to the device. Do you think an approach like this would cause server load issues? I did some digging on expected requests per second, and I think the most accurate dashboard for this would be this granfana link:

https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?orgId=1&viewPanel=12&from=now-7d&to=now&var-dc=thanos&var-site=codfw&var-service=mobileapps&var-prometheus=k8s&var-container_name=All

If you filter the endpoint by --domain_v1_page_mobile-html_--title_--revision-_--tid, which is our article html endpoint when viewing an article, we get this. The maximum is about 1.40K requests per second from the past 7 days. Note that this amount includes both iOS and Android apps, so I expect it to be lower as long as only iOS fetches these topics.

The linked Grafana dashboard shows uncached mobileapps traffic ranging between 300 rps and 1000 rps, but not the uncached traffic. Using Turnilo, I searched for WikipediaApp* user agents, and the total app traffic is around 5000–7000 rps (noting the Turnilo dashboard is sampled by 128). This aligns with the Turnilo dashboard you linked earlier.

I tried to do a rough approximation of what traffic we could exect. When examining total traffic to the MediaWiki API (/w/api.php), the range is 7000–10,000 rps, with roughly 10% coming from WikipediaApp: https://w.wiki/CEDu.

It's hard to determine the exact relationship between app page views and mobileapps requests. A single app page view likely generates multiple requests, not all requiring category lookups. Some of these could be cached at the frontend, reducing backend API load. For instance, many users might read the same articles, resulting in repetitive requests being served from cache rather than hitting the MediaWiki backend API.

But in the worst-case scenario, if every WikipediaApp request triggers a new API call for category lookups, total API traffic could increase by 50%, likely overloading the MediaWiki API. However, a more realistic estimate is that each app page view generates around a dozen requests, so only fraction requiring new category lookups. This could lead to a 5–10% increase in total API traffic, effectively doubling API calls from the app alone, which is still significant.

Given these rough estimates, I have reservations about implementing a feature where topics/categories are fetched on each article view. Maybe my assumptions about the number of requests are off. Do you have the chance to capture how many requests a single page view typically generates? I’ll investigate if querying multiple topics/categories at once via the provided cirrusdoc link might be more efficient. However, stuffing API calls in the end of the year would still create a substantial load in December.

Regarding the cirrusdoc API, it seems to return numerous properties beside the categories. May I ask why you are using it? Would it be feasible to query directly for categories? For example: https://en.wikipedia.org/w/api.php?action=query&format=json&titles=Cat&prop=categories. This might simplify and reduce the cost of calls. I guess because cirrus allows to query multiple page categories at once?

Additionally, could category/topic data be integrated into an existing API? The app already performs several API calls and calls to mobileapps, which mostly returns HTML afaik. Adding categories there might be challenging, but perhaps another frequent API call could be extended to include category information. Are there any specific APIs or services the app already uses that might be a suitable for this enhancement?

@Jelto

It's hard to determine the exact relationship between app page views and mobileapps requests. A single app page view likely generates multiple requests, not all requiring category lookups. Some of these could be cached at the frontend, reducing backend API load. For instance, many users might read the same articles, resulting in repetitive requests being served from cache rather than hitting the MediaWiki backend API.

Yep true, we can do that. We'll check to see if we already have the topics locally first before making the call.

Do you have the chance to capture how many requests a single page view typically generates?

I see 4 API calls that go out when viewing an article:

1. Get the user's watched status:

https://en.wikipedia.org/w/api.php?action=query&errorformat=html&errorsuselocal=1&format=json&formatversion=2&inprop=watched&prop=info&titles=Wicked%20(2024%20film)

2. Fetch the article html:

https://en.wikipedia.org/api/rest_v1/page/mobile-html/Wicked_(2024_film)

3. Fetch the article summary:

https://en.wikipedia.org/api/rest_v1/page/summary/Wicked_(2024_film)

4. Fetch related articles info for footer:

https://en.wikipedia.org/w/api.php?format=json&formatversion=2&origin=*&action=query&prop=pageimages%7Cdescription&piprop=thumbnail&pithumbsize=160&pilimit=3&generator=search&gsrsearch=morelike%3AWicked_%282024_film%29&gsrnamespace=0&gsrlimit=3&gsrqiprofile=classic_noboostlinks&uselang=content&smaxage=86400&maxage=86400

Regarding the cirrusdoc API, it seems to return numerous properties beside the categories. May I ask why you are using it? Would it be feasible to query directly for categories? For example: https://en.wikipedia.org/w/api.php?action=query&format=json&titles=Cat&prop=categories. This might simplify and reduce the cost of calls. I guess because cirrus allows to query multiple page categories at once?

We actually aren't interested in categories at this time, instead we want topics, which are generated via ORES. Some related docs are https://www.mediawiki.org/wiki/ORES#Topic_routing, https://www.mediawiki.org/wiki/ORES/Articletopic and https://wikitech.wikimedia.org/wiki/ORES#Guide_to_migrate_from_ORES_to_Lift_Wing. That cirrusdoc API call was just pulled from an old prototype we made demonstrating topics, but there very well could be a better API call to make for this instead. This curl request only returns topics for an article. We could make this when viewing an article after the article summary endpoint completes, so that we have a revision ID to send in:

curl https://api.wikimedia.org/service/lw/inference/v1/models/enwiki-articletopic:predict -X POST -d '{"rev_id": 1257586282}'

Additionally, could category/topic data be integrated into an existing API?

Yes, another thing we have discussed doing is incorporating topics into the article summary endpoint response: https://en.wikipedia.org/api/rest_v1/page/summary/Wicked_(2024_film). We already make this call upon viewing an article, and it is cached via RESTBase. This may be the best path forward, so that topics are cached across users.

@Jelto

We thought of a couple of other options for an app-side call (these are in order of my preference).

1. Add topics to watched status call

We already fetch the user's watched status upon article load (regardless of login status). What if we also ask for article topics in that same call? Would there still be load concerns if the app is making the same number of calls to the MediaWiki API?

We would fetch:

https://en.wikipedia.org/w/api.php?action=query&format=json&prop=cirrusdoc%7Cinfo&titles=Cat&formatversion=2&cdincludes=weighted_tags&inprop=watched

upon article load instead of:

https://en.wikipedia.org/w/api.php?action=query&format=json&prop=info&titles=Cat&formatversion=2&inprop=watched

Regarding the cirrusdoc API, it seems to return numerous properties beside the categories.

Note I found the cdincludes parameter so that we only return topics and not all possible properties.

2. Fetch topics once per day

Alternatively, we could have the app fetch and persist article topics only once at the end of each day for the articles they read that day. This way the requests aren't happening as one big dump at once at the end of the year, but aren't happening as often as every page view either. Maybe this middle-ground approach might help?

Let me know what you think!

Thanks for digging into the cdincludes parameter option! Bundling the API calls into one indeed seems like a good idea for better performance, especially if the result is cached for the most frequently accessed sites.

I'm not very familiar with Cirrus and ORES either, but based on the docs, it looks like the weighted_tags are stored within Cirrus and Elasticsearch. This suggests no dedicated API call (e.g., to ORES) would be required. Ideally, for uncached requests, it should still be the same Elasticsearch API call, but I will confirm this with the search team. I’ll also ask if adding the cdincludes parameter could cause any unexpected issues with Cirrus.

Regarding ORES, the docs do indicate a lot of deprecation in favor of LiftWing. This means we need to confirm whether the weighted_tags you’re using will still exist in the future. Which specific tags were you planning to use? Were they the classification.ores.articletopic ones?

I also noticed your curl example uses the enwiki-articletopic prediction model. Would the "year in review" feature need to be available for other languages besides English Wikipedia? If so, we might want to explore more language-agnostic approaches.

The LiftWing docs mention the outlink-topic-model as a language-agnostic option for article topics. Here's an example call:

# English:
curl https://api.wikimedia.org/service/lw/inference/v1/models/outlink-topic-model:predict -X POST -d '{"page_title": "Cat", "lang": "en", "threshold": 0.5}' -H "Content-type: application/json"
{"prediction":{"article":"https://en.wikipedia.org/wiki/Cat","results":[{"topic":"STEM.STEM*","score":0.9890230894088745},{"topic":"STEM.Biology","score":0.9808859825134277}]}}
# German:
curl https://api.wikimedia.org/service/lw/inference/v1/models/outlink-topic-model:predict -X POST -d '{"page_title": "Katze", "lang": "de", "threshold": 0.5}' -H "Content-type: application/json"
{"prediction":{"article":"https://de.wikipedia.org/wiki/Katze","results":[{"topic":"STEM.STEM*","score":0.9173126816749573},{"topic":"STEM.Biology","score":0.7371681928634644}]}}

I can reach out to the Machine Learning team to confirm whether ORES tags will still be available in the next 1+ years, and if not, which LiftWing API call might be a better fit. I’ll also ask about the load capacity of the relevant endpoints to ensure the service can handle the required volume.

@Jelto

Regarding ORES, the docs do indicate a lot of deprecation in favor of LiftWing. This means we need to confirm whether the weighted_tags you’re using will still exist in the future. Which specific tags were you planning to use? Were they the classification.ores.articletopic ones?

Yes, those containing articletopic was the plan.

I did find some documentation (view last paragraph) that indicates there's caching in ORES but not LiftWing. So would be worth checking into further.

I also noticed your curl example uses the enwiki-articletopic prediction model. Would the "year in review" feature need to be available for other languages besides English Wikipedia? If so, we might want to explore more language-agnostic approaches.

Thanks for catching that. Yes, we would want it to work for other languages as well. So maybe we can try your alternative curl requests instead if we decide to make a dedicated separate call.

I can reach out to the Machine Learning team to confirm whether ORES tags will still be available in the next 1+ years, and if not, which LiftWing API call might be a better fit. I’ll also ask about the load capacity of the relevant endpoints to ensure the service can handle the required volume.

That would be great, thank you! Please let us know what you find out.

Hi @Jelto, checking back in on this. I'd like to know if pulling the weighted_tags object in the MW API from the client will work from a caching and deprecation standpoint. Failing that if a call to the LiftWing API would work. My impression is that the LiftWing API from the app won't work due to a lack of caching there. That's based on feedback I've heard from @Seddon, please correct me if I'm wrong! Thanks.

Adding another angle to expected requests per second:

Per https://phabricator.wikimedia.org/T379119#10434052, I noted that we already fetch their watched status upon article view (whether they are logged in or not). When I look up MW API calls specifically from the iOS app, matching the watched status call:

https://w.wiki/DJup

The max in the last 7 days is 3.3K requests per hour. Divide that by 3600, then multiplied by the 128 sampled = (3300 / 3600) * 128 = 117.333 requests per second. So we can expect that many calls to a LiftWing API, if we make it from the client.

Also note that we are fetching their watched status in the editor flow and on our revision diff view, not just article view. Plus we can cache client-side so we aren't fetching topics for repeat articles. So it would likely be a little lower.

I think this might be more accurate than checking all calls against the MW API as a whole, for both apps (https://phabricator.wikimedia.org/T379119#10366132).

I see that you are interested in using the Search API, in particular using CirrusDoc. CirrusDoc is exposing the internal representation of a Search Document, and isn't meant to be stable or reused outside of debugging. Search is also probably not the right place to get additional information about a set of known articles. It is really meant to run queries and retrieve a set of results. See https://wikitech.wikimedia.org/wiki/Search/Technical_interactions#Limitations for slightly more context.

All that being said, we might be able to create another way to access article features, in an efficient and reliable way. It might make sense to have a chat!

Note that this need of accessing peripheral or computed features of specific articles has been raised a few times and it would probably be a good idea to address it in a generic way (second order dependency).

I reached out to the machine learning team (see full thread in Slack). Key takeaways:

• The ORES models like classification.ores.articletopic are served by LiftWing, so ORES deprecation doesn’t affect them and the name is just a bit confusing.
• They recommend using the outlink-topic-model over the old enwiki-articletopic.

Also, the estimated traffic of 60–120 rps is likely too high for the LiftWing endpoints. Their suggestion is to use the MediaWiki API and CirrusSearch instead (which was one of the first ideas in this task: T379119#10332203).

As far as I understand, this conflicts with @Gehel’s suggestion. Or is CirrusDoc/Search API different from https://en.wikipedia.org/w/api.php?prop=cirrusdoc...? @Gehel?

In T379119#10619055, @Jelto wrote:

As far as I understand, this conflicts with @Gehel’s suggestion. Or is CirrusDoc/Search API different from https://en.wikipedia.org/w/api.php?prop=cirrusdoc...? @Gehel?

Those are the same things, and cirrusdoc should really not be used here.

The issue with cirrusdoc is that it is not a stable api with any guarantees around shape. cirrusdoc is a debug api that exposes the internal representation of a wiki page within the search engine. It changes based on what externally would look like arbitrary decisions made within search. The cirrusdoc api was initially implemented as a way for integration testing to know when an edit had made it into the search indices and could then attempt to perform queries based on the edit. It can also be useful when debugging issues in production to see what the search index knows about a page, to help understand why a page might not have been returned for a particular search. We don't want to get into a situation where the internal representation is locked in place because external consumers depend on specific shapes, we want to keep those limitations internal to the search infrastructure.

As a debug endpoint it doesn't have any particular optimizations around performance. If we ask it for 50 docs in one api call it will make 50 sequential calls to the backend servers to fetch those (fixable, but was never important as that isn't it's purpose). The way data is represented isn't designed to be understood by anyone outside search. Since the original introduction of ML generated topic's for articles they have been represented at least three different ways here. For example the response says classification.ores.articletopic, but really it's serving up predictions from the outlink-topic-model. The prefix hasn't been important to change within search because the prefix isn't exposed to users, the prefixes are an internal implementation detail of search that we didn't change when switching between models but may change some day in the future.

It is entirely plausible that an api could be created to report the topics. It could, at least initially, source the data from the search engine representation. But there is an open question about if search is even the right place for this kind of thing. This is essentially transforming the search engine into a key/value store which is technically possible, but it's not clear that is an appropriate use case.

@HNordeenWMF I explored a local ML model for article topics. This was trained based on the article outlinks, similar to our server-side model. Training involved pulling 500 articles per topic and getting their first 500 outlinks (wikidata Q-item IDs that are linked in the article). The resulting model was 6MB, so the size is acceptable for bundling into the app. There are some inaccuracies - this was just some early work to see how feasible it is. It's possible we can get it to be more accurate, but more time and exploration would be needed for that.

I pushed a prototype to Experimental #240. This takes the last 500 articles in History, fetches their outlinks, runs the result through the model and then lists the articles by topic. Performance is slow on this screen - the idea would be to fetch outlinks in an ongoing basis upon article view rather than all at once like this, though the question here would be if that would cause the same server load concerns as fetching the original server-side article topics upon article view.

I think the result here is inconclusive, but I just wanted to document the alternative approach.

Oh hm! Did you know that since 2023 outlink topic predictions are available in an event stream in Kafka and a Data Lake (Hive) table?

The outlink topic prediction stream is not available publicly at https://stream.wikimedia.org, but making it so is a simple config change (like this one).

Now you just need a view of this data for pages that you can query in bulk, eh?

Hi @Ottomata! Thanks for chiming in. The app would need to be able to fetch topics for a user's read articles (either individually or in bulk) via an external API. LiftWing does have an endpoint for that:

curl https://api.wikimedia.org/service/lw/inference/v1/models/outlink-topic-model:predict -X POST -d '{"page_title": "Cat", "lang": "en", "threshold": 0.5}' -H "Content-type: application/json"
{"prediction":{"article":"https://en.wikipedia.org/wiki/Cat","results":[{"topic":"STEM.STEM*","score":0.9890230894088745},{"topic":"STEM.Biology","score":0.9808859825134277}]}}

But it's been determined that this endpoint cannot handle the server load that the apps would bring by calling this upon each article view. Is there an external API to pull this event stream / data lake table information? Or if not, would it be easy to spin one up, and would we run into the same server load constraints if we are fetching this information from the app?

Removing Search Platform as it seems that we're not needed.

there an external API to pull this event stream / data lake table information? Or if not, would it be easy to spin one up, and would we run into the same server load constraints if we are fetching this information from the app?

event stream: we could expose it at stream.wikimedia.org like other streams. But, I don't think this will work well for your use case. This service is meant for low volume tool developers. Connecting all installed apps to it would be bad.

Data lake: unfortunetly not in a standard way.

What you are asking for is similar to T341649: Provide an easy way for MediaWiki to fetch aggregate data from the data lake. More generally:

• Derived Data to Support Production Features | Product Brief.
• Better access paths for LiftWing data in MediaWiki+

Data Gateway might help, but IIUC there is no prioritization of this work in the APP. We keep trying to get ahead of the game to call out the need for what you are asking for. If you'd like to see something like this prioritized, please bubble it up the manager chain!

Ah! Apologies! I'm just back from leave and am catching up, and just learned that these discussions are in progress! Thank you!