Added answers to our questions in the description above. I do have a couple of questions for the notifications API team maintainers:

Question 1

I would like to understand further what the significance of the wiki project that we're making the API call on and the notwiki values, and how it affects the results. I have seen some unexpected results (these links will work differently based on account, but I'll summarize my account results):

https://en.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=*&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://www.mediawiki.org/w/api.php?action=query&meta=notifications&notwikis=*&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://es.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=*&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model

EN Wikipedia only has enwiki notifications, whereas MediaWiki notifications has enwiki notifications plus mediawikiwiki notifications. Similar issue with ES Wikipedia, it has eswiki notifications in the results along with enwiki notifications. At the same time, EN Wikipedia has a continue value, but the continue value for ES and MediaWiki wikis calls are null. So how does notwikis=* work? I would have expected all of these responses to be identical.

https://en.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=enwiki|eswiki|mediawikiwiki&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://www.mediawiki.org/w/api.php?action=query&meta=notifications&notwikis=enwiki|eswiki|mediawikiwiki&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://es.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=enwiki|eswiki|mediawikiwiki&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model

These are more similar, but still the call to es.wikipedia.org and mediawiki.org have null continue values.

Question 2

Is the notification ID for each object in the response unique across all wiki projects? Or only within that wiki?

@cmadeo Just a heads up about design considerations we might need to make based on the approach I want to try.

1. Similar to Android, I want to trigger a bulk import of notifications (for iOS the first time the land on the center), which will make filtering later on way easier for us. (See question 4's answer above) - we may want to give some sort of indication to the user that we are importing (it doesn't have to block the whole screen, they should be able to browse local data as it comes in). And we might want to hide the counts label or give some indication that it's not fully up to date yet as we're importing. This should be a one-time thing per install, subsequent visits to the center will be much faster since it's a more basic refresh.

2. The answer to question 6 also is a little iffy since we are limited to 50 (or 500 can be arranged I'm sure) notifications to mark as unread or read. We can try splitting up the calls so maybe no design change will need to be taken, stay tuned.

3. I also wanted to point out that the Notifications API differentiates between the notifications area being "seen" vs "read". We can flag the center as a whole (or split up by all alerts or all messages, no further) as "seen". The notifications will still be considered unread since they didn't tap the individual notification, but I believe this flag is used on Desktop as the extra badge color highlight you see when there's a new notification that you've never seen in the center before. Then on Desktop once you display the center for the first time, the badge changes from color to grayed-out. Just letting you know you have this ability in case you want to lean on it. We haven't discussed badging much yet, but this would be the distinction between our badges meaning "you have unread notifications that you've seen and never tapped" and "you have unread notifications that you've never seen".

^ The "2" here can be blue if they've never been seen.

Thanks @Tsevener ! These are all super helpful updates!

1. Similar to Android, I want to trigger a bulk import of notifications (for iOS the first time the land on the center), which will make filtering later on way easier for us. (See question 4's answer above) - we may want to give some sort of indication to the user that we are importing (it doesn't have to block the whole screen, they should be able to browse local data as it comes in). And we might want to hide the counts label or give some indication that it's not fully up to date yet as we're importing. This should be a one-time thing per install, subsequent visits to the center will be much faster since it's a more basic refresh.

Sounds good! Very happy to hide the counts until everything comes in and it makes sense for us to have a loading/importing state. I'm curious what we can borrow/build off of from reading list imports. I'll start working on some early design ideas

2. The answer to question 6 also is a little iffy since we are limited to 50 (or 500 can be arranged I'm sure) notifications to mark as unread or read. We can try splitting up the calls so maybe no design change will need to be taken, stay tuned.

Good to know, also happy to work within this constraint and think about how we can modify the selection state (eg. maybe instead of 'select all' it's 'select first 50')

3. I also wanted to point out that the Notifications API differentiates between the notifications area being "seen" vs "read". We can flag the center as a whole (or split up by all alerts or all messages, no further) as "seen". The notifications will still be considered unread since they didn't tap the individual notification, but I believe this flag is used on Desktop as the extra badge color highlight you see when there's a new notification that you've never seen in the center before. Then on Desktop once you display the center for the first time, the badge changes from color to grayed-out. Just letting you know you have this ability in case you want to lean on it. We haven't discussed badging much yet, but this would be the distinction between our badges meaning "you have unread notifications that you've seen and never tapped" and "you have unread notifications that you've never seen".

Oh wow! I hadn't noticed this on web! Hmm, this might be helpful with badging! We could follow the same convention and down play the color highlight in the badge when the notification has been seen but not read.

It seems like the API only filters on read/unread, so filtering by notification types and projects and searching will need to be on-device. (API is also paged. Filtering + paging will be tricky).

If you're looking to filter by notification type, it wouldn't be that hard to add that to the Echo API. The filter for alerts vs messages (notsections) already does that behind the scenes.

So how does notwikis=* work?

I'm not sure if this helps to explain the behavior you saw, but notwikis=* means "all wikis where I have unread notifications, plus the local wiki". So if, for example, you have unread notifications on enwiki but nowhere else, then using notwikis=* in an API call to en.wikipedia.org will only show enwiki notifications; in an API call to mediawikiwiki it'll show mediawikiwiki and enwiki notifications, etc.

Is the notification ID for each object in the response unique across all wiki projects? Or only within that wiki?

Notification IDs are unique only within each wiki. As you suspected, the combination of the notification ID and the wiki name is unique. The one exception to this rule is that notifications with negative numbers are "fake"; but this only happens if notcrosswikisummary=1 is passed, in which case the top notification will be a fake one (whose ID is -1) with information about how many unread notifications of what type you have on foreign wikis. It's what powers this feature on the web:

I think a one-time bulk import is better

I think that will work fine, especially if it already works on Android, but if you're using the cross-wiki features I would caution you to be careful about the fact that, if a user no longer has any unread notifications on a wiki, that wiki will stop showing up in notwikis=* (and vice versa, you might discover a new wiki later when the user first has an unread notification there). Relatedly, I'll point out that the updates you're looking for aren't only new notifications, but also existing notifications being marked as read (or as unread!) on another platform.

Could we populate the “123 messages from 6 projects” label from the API (particularly if this is paged)?

Yes you can, using the notcrosswikisummary parameter I described above. The fake notification that is returned by this (with id=-1 and type=foreign) has a count property with the number of unread foreign notifications, and a sources property with a list of wikis where those notifications live.

Can we get the list of projects a user is “in” for Echo, or do we just know about them as they arrive locally?


As I said before, Echo itself only really tells you about wikis where the user has unread notifications at the time of your request. There is another API for finding out on which wikis the user has an account, so you could use that instead, but be warned that that list can be very long, since it'll be just about every wiki the user has ever visited, even if they've never done anything there. My volunteer account exists on 279 wikis, your work account exists on 320, and one of our more prolific coworkers is on 570. There are 976 wikis in total at present (some of which are private or closed, so the theoretical maximum that you could get back from this API is probably about 900-950). That is all to say that making API requests to all of a user's wikis separately might not be feasible. With the notwikis parameter you can batch up to 50 wikis at a time, so that might work if you're willing to make a half dozen requests or so (plus more to paginate it out when there are a lot of notifications, but you were already planning that I think). So instead you may want to have the user configure this list themselves, or use the top ~50 wikis where they have the most edits, or add a wiki to the list when you discover an unread notification on it through the cross-wiki feature, or some combination of those things, to keep the list of wikis you're checking to a more manageable size.

The answer to question 6 also is a little iffy since we are limited to 50 (or 500 can be arranged I'm sure) notifications to mark as unread or read. We can try splitting up the calls so maybe no design change will need to be taken, stay tuned.

The "500 for certain clients" thing is based on user rights (tied to the user account), so with the current API infrastructure we can't privilege the apps to get higher limits. (The feature was meant for bots that have their own account.) Splitting up the calls is what I would recommend.

I also wanted to point out that the Notifications API differentiates between the notifications area being "seen" vs "read".

This whole paragraph is an excellent description of how the "seen" feature works and how it differs from "read", you got it exactly right. When we worked on this, our team had trouble wrapping our heads around this and everyone was always confused when reasoning about it, but you figured out exactly how it works and wrote a better explanation of it than I did at the time. That's some impressive reverse engineering.

@Catrope wow, thank you so much for all of this! It explains everything and corrects a lot of assumptions I was making. I had the subdomain perspective and the way it affects things all wrong but I think now I get it.

If you're looking to filter by notification type, it wouldn't be that hard to add that to the Echo API.

That actually has been scrapped in our latest design mocks, but it's good to know it may not be too bad to have API support if that makes it back in for a v2.

I'm not sure if this helps to explain the behavior you saw, but notwikis=* means "all wikis where I have unread notifications, plus the local wiki".

Yes, that's why I was seeing that behavior, I only had unread notifications on EN so the other projects with read notifications were filtered out. If I pass in notwikis=enwiki|mediawikiwiki, I do see read and unread from both projects, but continue is null if I'm calling from mediawiki.org because I have very few mediawiki notifications, even though there are more pages of EN notifications. That's the perspective part that threw me off - I wouldn't be able to populate read and unread data from multiple projects in a single call, I think I'll need to break them up into en.wikipedia.org?notwikis=enwiki and mediawiki.org?notwikis=mediawikiwiki during the import phase for the continue IDs to work out.

So instead you may want to have the user configure this list themselves, or use the top ~50 wikis where they have the most edits, or add a wiki to the list when you discover an unread notification on it through the cross-wiki feature, or some combination of those things, to keep the list of wikis you're checking to a more manageable size.

Yep, we have a list of app languages that the user already has set that we can lean on, and are also planning on having a filter by project feature for these notifications. Adding the projects returned in notwikisummary is a great idea too, thanks for the heads up about that.

This whole paragraph is an excellent description of how the "seen" feature works and how it differs from "read", you got it exactly right.

Thanks! The leap from seen value to badge color was guesswork on my part, so it's good to know sometimes I get it right! 😆

In T284239#7233171, @Tsevener wrote:

Yes, that's why I was seeing that behavior, I only had unread notifications on EN so the other projects with read notifications were filtered out. If I pass in notwikis=enwiki|mediawikiwiki, I do see read and unread from both projects, but continue is null if I'm calling from mediawiki.org because I have very few mediawiki notifications, even though there are more pages of EN notifications. That's the perspective part that threw me off - I wouldn't be able to populate read and unread data from multiple projects in a single call, I think I'll need to break them up into en.wikipedia.org?notwikis=enwiki and mediawiki.org?notwikis=mediawikiwiki during the import phase for the continue IDs to work out.

You've found a bug (two bugs, really) in the limit/continue handling here that I hadn't realized existed. I don't think anything in the UI uses multiple notwikis values, which might be why we never noticed this. These are almost certainly my fault, because I wrote this code in 2016 and it hasn't changed much since.

1. notlimit limits the number of results per wiki, not the total number of results returned. My account has 5 notifications on hewiki and 42 notifications on enwiki, so you'd expect notwikis=hewiki|enwiki&notlimit=20 to return 20 results (maybe 5 from hewiki and 15 from enwiki, or some other mix depending on how old they are). But instead, it returns 25 results (5 from hewiki and 20 from enwiki) because the limit is applied to each wiki separately. This is probably not a big problem, but it is surprising that you could set notlimit=20 and potentially get 100+ results if you listed 5+ wikis (either explicitly, or implicitly with notwikis=*)
2. The returned continue value is based entirely on local notifications, and doesn't take foreign notifications into account at all. This means that, in the previous example (notwikis=enwiki|hewiki&notlimit=20), I get a continue value back if I send the request to en.wikipedia.org (because there are more than 20 notifications there) but if I send the same request to he.wikipedia.org I get continue: null, because I've gotten all hewiki notifications, even though there are more enwiki notifications. But the notcontinue parameter does affect the foreign notifications returned in the next request, which means that if you do get a continue value, using it will either skip or duplicate foreign notifications in your pagination.

In brief, paginating through notifications from foreign wikis is completely broken. I think #2 is fixable (by cleverly proxying through the continue value from the foreign API request that we make internally), but it would require some work and then careful testing.

@Tsevener I'm just catching-up on the conversations happening here in the comments (big thanks to Roan for all of your help!)

I'm curious if you think that it'd be better/easier for us to scrap the by-project inboxes for v1?

@cmadeo I think there's a couple of directions we could go in -

1. I think the easiest would be displaying notification center on a per-wiki basis, since the local wiki pagination works in the API. So that would be keeping the project inbox screen, but treating it as single-select that applies to the whole in app notification center. We couldn't treat projects as an inbox filter that could be removed to show more notifications in a list combined with notifications from other projects.
2. A second option to try would be for us to fetch notifications from all the different wikis separately under-the-hood, and display them combined in a single list. The filter will act as a local filter only on data we have already fetched and persisted (which I was already leaning towards). This method isn't totally uncharted territory - we do something similar for the Explore feed because all of that data is fetched from their respective wikis and combined rather than a single API call. We could keep our design with this method.

Regardless of which option we choose, I am starting to feel like we should scrap the bulk import. Scrapping the bulk import would just mean we would want to change the counts text at the bottom from "[n] notifications in [n] projects" to "[n] unread notifications in [n] projects". The unread number across projects is something that can be easily fetched in a single call from the server.

Thanks for this update and clarification @Tsevener! I like the second approach (seems closest to the current design proposal 😉), but happy to make changes as things come up.

Also I'll update the mocks to include the 'unread' element in the thread. Thanks!

Hi @Catrope, thanks for the heads up about the bugs! We're still talking it over on our side how to adjust.

I'm not sure if you're the right person to ask on this (feel free to tag someone else if not), but do you think if we broke it up into separate calls (en.wikipedia.org?notwikis=enwiki, es.wikipedia.org?notwikis=eswiki, etc.) for each app language the user has set, the server would be able to handle the load? Particularly if we decide to do a bulk import, so we'd be fetching N number of projects, paged to the beginning of notification time in one go. Android bulk imports notwikis=*, so this bulk import method would be a different beast than theirs since we'd like both read and unread notifications from all app language wikis they have set up.

In T284239#7237568, @Tsevener wrote:

Hi @Catrope, thanks for the heads up about the bugs! We're still talking it over on our side how to adjust.

I'm not sure if you're the right person to ask on this (feel free to tag someone else if not), but do you think if we broke it up into separate calls (en.wikipedia.org?notwikis=enwiki, es.wikipedia.org?notwikis=eswiki, etc.) for each app language the user has set, the server would be able to handle the load? Particularly if we decide to do a bulk import, so we'd be fetching N number of projects, paged to the beginning of notification time in one go.

I'm not totally the right person to ask, but I'm also not sure that that right person exists :) . I think what you propose will be fine, especially since I'm guessing that the number of app languages users typically set is low (I imagine it's rarely higher than 10). A bulk import would also happen only once per user. For updating the app with subsequent notifications, how often would you make API requests, and would those be for all wikis or only the selected one?

(Also note that the notwikis parameter is optional and defaults to the current wiki, so in your example you could omit it.)

@Catrope yeah for subsequent notifications, we'd fetch the first page for each wiki and only page further if it seems like there's more than one page of new stuff. It would be ideal to do this whenever they entered the app notification center, foregrounded on the app notification center, received a push while in the app notification center, or pulled to refresh in the app notification center.

To try to reduce the number of calls we could add a minimum time passed since last refresh rule for the first two. The user can also have a local filter set for projects, so we can only refresh the ones they have set to display. And when we receive a push, if we can determine which project the push is from, we can only refresh that one.

I also feel like we should trash the local data and re-do the bulk import if the user logs out and logs back in.

(Also note that the notwikis parameter is optional and defaults to the current wiki, so in your example you could omit it.)

Ah I missed that, thanks!

OK, that sounds fine. To reduce concerns about server load, you could also make these requests in series rather than in parallel, if you think that won't degrade the user experience.

Thanks for looking into this. If I am understanding this correctly, from the Android perspective we don't need to make any changes at this time.

hi @Catrope, I have a quick question about notcrosswikisummary:

Could we populate the “123 messages from 6 projects” label from the API (particularly if this is paged)?

Yes you can, using the notcrosswikisummary parameter I described above. The fake notification that is returned by this (with id=-1 and type=foreign) has a count property with the number of unread foreign notifications, and a sources property with a list of wikis where those notifications live.

In my digging I noticed the summary object only seems to show in the response if the user has their "Cross-wiki notifications/Show notifications from other wikis" preference checkmarked, otherwise I can't get their global unread counts. Is that expected? I suppose I could always try writing to that preference value via the API before requesting with notcrosswikisummary=1, but I wanted to double-check first before going down that path.

API reference: https://en.wikipedia.org/w/api.php?action=query&meta=notifications&notcrosswikisummary=1&notlimit=1
Screenshot:

Based on reading the code it looks like this was intended. When the user opens the notification dropdown, we make an API request that always sets notcrosswikisummary=1, regardless of the user's preference, and rely on the API to check the preference and return the correct result. (On Special:Notifications we don't set the notcrosswikisummary parameter, but that's because we want local notifications only there, because of how that UI is laid out per-wiki.)

However, this behavior is undocumented and not intuitive, and I would be in favor of changing it. I think the best way to do that would be to change the API parameter to take three values, so you can choose between not getting the cross-wiki summary, or definitely getting it, or only getting it if the preference is set.