Page MenuHomePhabricator

[Spike]: App Notification Center Read/Write Questions
Closed, ResolvedPublicSpike

Description

This spike is for engineers to answer some questions we have around reading notifications in the app notification center:

Question 1

It seems like the API only filters on read/unread, so filtering by notification types and projects and searching will need to be on-device. (API is also paged. Filtering + paging will be tricky).

  • We could we fetch and persist all notification history right away, which would simplify filtering and searching.

Answer: Correct, the API only filters on read and unread. But it wouldn't be difficult to add that ability per https://phabricator.wikimedia.org/T284239#7231144.

Sidenote
The notifications API response can change depending on which project we're fetching from. notwikis=* returns only unread notifications from foreign wikis, and listing all projects out (notwikis=enwiki|mediawikiwiki...) only pages against the local wiki (i.e. continue can be null even if there are more notifications to see on a foreign wiki), I think the calls will need to be separated out per project, and the results combined to get the single list view of multiple projects with both read and unread from each. See the thoughts around https://phabricator.wikimedia.org/T284239#7231144 and https://phabricator.wikimedia.org/T284239#7233171 for a reference.

Question 2

What sort of identifiers does it return and could we use that for proper syncing (i.e. marking something as read if it was read on desktop)?

Answer: Notifications are only unique to each wiki (https://phabricator.wikimedia.org/T284239#7231144).

Question 3

Prototype the fetching and displaying of notifications in OS notification content vs app list screen. Be able to distinguish a new notification for display in OS notification vs one to display with an unread indicator in the app notification center.

Answer: This is done in the push-notifications-prototype branch. With this approach I tried to only fetch deep into the user's notification history as they reached the bottom of the screen, but this felt prone to bugs with filtering and the counts label. I think a one-time bulk import is better, Android has already proved the approach (they do it upon install), and there's been no server load complaints. I suggest doing it the first time they visit the center so that we aren't fetching the history for someone that never intends to look at notifications.

For a new notification to display in the OS notification, I propose we fetch directly from the notification service extension (they will likely have a connection having just received the notification, and I want to avoid more app container database deadlock issues). I don't think there's a case where we would want to display a push notification that has been marked as read, so in this instance a single call leaning on notwikis=*&notfilter=!read could get us the data we need.

Question 4

Could we populate the “123 messages from 6 projects” label from the API (particularly if this is paged)?

  • Is it okay if that reflects the local counts and not the server counts?
  • Are we only keeping new notification locally or are we recreating Echo inbox historically? If so how far back? If not how do educate users?

Answer: We can populate this from the server if we're only concerned with unread count. (https://phabricator.wikimedia.org/T284239#7231144). Asking for notcrosswikisummary=1 will give us a list of foreign wikis that have unread notifications, and asking for notprops=list|count will give us a global count of unread notifications.

If we're doing a bulk import we could also base this count on local numbers, but these numbers could be way off base during the import. We could still show the numbers during the import and have them update on the fly as we get more data, or hide the label until the initial bulk import completes. I also suggest we maybe have some sort of rotating syncing indicator somewhere so the user knows we're doing stuff.

Question 5

Can we get the list of projects a user is “in” for Echo, or do we just know about them as they arrive locally?


Answer: See https://phabricator.wikimedia.org/T284239#7231144 for answers. I suggest we determine what wiki projects to fetch from based on their app languages, plus extras like commonswiki and wikidatawiki. We can also make an initial call with notcrosswikisummary=1 to get any additional wiki projects that have unread notifications that aren't already in their app languages. We can use this same logic to populate the projects filter options, or base those options on the local notifications data we have imported.

Question 6
  1. Does API support the edit actions from an API perspective? (mark as read, deselect all, mark as unread)?
    • Is it marking only items on screen or all items on server (including those that haven’t been fetched yet) as read?

Answer: Yes, see action=echomarkread section here. It looks like it supports marking a particular notification or list of notifications as read, marking all notifications as read, and marking a particular notification or list of notifications as unread (no mark all as unread option). Marking all notifications as read would include notifications that may not have been fetched from the server yet. There's a hitch where sending in a list of notifications to mark as read is limited to 50. If the user selected all 100 notifications on their screen and chose mark and read or unread, we'll need to split up the call into two calls of 50 IDs.

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptJun 3 2021, 4:31 PM

Added answers to our questions in the description above. I do have a couple of questions for the notifications API team maintainers:

Question 1

I would like to understand further what the significance of the wiki project that we're making the API call on and the notwiki values, and how it affects the results. I have seen some unexpected results (these links will work differently based on account, but I'll summarize my account results):

https://en.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=*&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://www.mediawiki.org/w/api.php?action=query&meta=notifications&notwikis=*&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://es.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=*&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model

EN Wikipedia only has enwiki notifications, whereas MediaWiki notifications has enwiki notifications plus mediawikiwiki notifications. Similar issue with ES Wikipedia, it has eswiki notifications in the results along with enwiki notifications. At the same time, EN Wikipedia has a continue value, but the continue value for ES and MediaWiki wikis calls are null. So how does notwikis=* work? I would have expected all of these responses to be identical.

https://en.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=enwiki|eswiki|mediawikiwiki&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://www.mediawiki.org/w/api.php?action=query&meta=notifications&notwikis=enwiki|eswiki|mediawikiwiki&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model vs
https://es.wikipedia.org/w/api.php?action=query&meta=notifications&notwikis=enwiki|eswiki|mediawikiwiki&notlimit=max&notprop=count|list|seenTime&notbundle=1&notformat=model

These are more similar, but still the call to es.wikipedia.org and mediawiki.org have null continue values.

Question 2

Is the notification ID for each object in the response unique across all wiki projects? Or only within that wiki?

@cmadeo Just a heads up about design considerations we might need to make based on the approach I want to try.

  1. Similar to Android, I want to trigger a bulk import of notifications (for iOS the first time the land on the center), which will make filtering later on way easier for us. (See question 4's answer above) - we may want to give some sort of indication to the user that we are importing (it doesn't have to block the whole screen, they should be able to browse local data as it comes in). And we might want to hide the counts label or give some indication that it's not fully up to date yet as we're importing. This should be a one-time thing per install, subsequent visits to the center will be much faster since it's a more basic refresh.
  1. The answer to question 6 also is a little iffy since we are limited to 50 (or 500 can be arranged I'm sure) notifications to mark as unread or read. We can try splitting up the calls so maybe no design change will need to be taken, stay tuned.
  1. I also wanted to point out that the Notifications API differentiates between the notifications area being "seen" vs "read". We can flag the center as a whole (or split up by all alerts or all messages, no further) as "seen". The notifications will still be considered unread since they didn't tap the individual notification, but I believe this flag is used on Desktop as the extra badge color highlight you see when there's a new notification that you've never seen in the center before. Then on Desktop once you display the center for the first time, the badge changes from color to grayed-out. Just letting you know you have this ability in case you want to lean on it. We haven't discussed badging much yet, but this would be the distinction between our badges meaning "you have unread notifications that you've seen and never tapped" and "you have unread notifications that you've never seen".

Screen Shot 2021-07-16 at 5.04.41 PM.png (43×76 px, 1 KB)

^ The "2" here can be blue if they've never been seen.

Thanks @Tsevener ! These are all super helpful updates!

  1. Similar to Android, I want to trigger a bulk import of notifications (for iOS the first time the land on the center), which will make filtering later on way easier for us. (See question 4's answer above) - we may want to give some sort of indication to the user that we are importing (it doesn't have to block the whole screen, they should be able to browse local data as it comes in). And we might want to hide the counts label or give some indication that it's not fully up to date yet as we're importing. This should be a one-time thing per install, subsequent visits to the center will be much faster since it's a more basic refresh.

Sounds good! Very happy to hide the counts until everything comes in and it makes sense for us to have a loading/importing state. I'm curious what we can borrow/build off of from reading list imports. I'll start working on some early design ideas

  1. The answer to question 6 also is a little iffy since we are limited to 50 (or 500 can be arranged I'm sure) notifications to mark as unread or read. We can try splitting up the calls so maybe no design change will need to be taken, stay tuned.

Good to know, also happy to work within this constraint and think about how we can modify the selection state (eg. maybe instead of 'select all' it's 'select first 50')

  1. I also wanted to point out that the Notifications API differentiates between the notifications area being "seen" vs "read". We can flag the center as a whole (or split up by all alerts or all messages, no further) as "seen". The notifications will still be considered unread since they didn't tap the individual notification, but I believe this flag is used on Desktop as the extra badge color highlight you see when there's a new notification that you've never seen in the center before. Then on Desktop once you display the center for the first time, the badge changes from color to grayed-out. Just letting you know you have this ability in case you want to lean on it. We haven't discussed badging much yet, but this would be the distinction between our badges meaning "you have unread notifications that you've seen and never tapped" and "you have unread notifications that you've never seen".

Oh wow! I hadn't noticed this on web! Hmm, this might be helpful with badging! We could follow the same convention and down play the color highlight in the badge when the notification has been seen but not read.

It seems like the API only filters on read/unread, so filtering by notification types and projects and searching will need to be on-device. (API is also paged. Filtering + paging will be tricky).

If you're looking to filter by notification type, it wouldn't be that hard to add that to the Echo API. The filter for alerts vs messages (notsections) already does that behind the scenes.

So how does notwikis=* work?

I'm not sure if this helps to explain the behavior you saw, but notwikis=* means "all wikis where I have unread notifications, plus the local wiki". So if, for example, you have unread notifications on enwiki but nowhere else, then using notwikis=* in an API call to en.wikipedia.org will only show enwiki notifications; in an API call to mediawikiwiki it'll show mediawikiwiki and enwiki notifications, etc.

Is the notification ID for each object in the response unique across all wiki projects? Or only within that wiki?

Notification IDs are unique only within each wiki. As you suspected, the combination of the notification ID and the wiki name is unique. The one exception to this rule is that notifications with negative numbers are "fake"; but this only happens if notcrosswikisummary=1 is passed, in which case the top notification will be a fake one (whose ID is -1) with information about how many unread notifications of what type you have on foreign wikis. It's what powers this feature on the web:

Screenshot from 2021-07-22 11-22-28.png (313×506 px, 39 KB)

But it sounds like you're not passing notcrosswikisummary=1, so this shouldn't affect you.

I think a one-time bulk import is better

I think that will work fine, especially if it already works on Android, but if you're using the cross-wiki features I would caution you to be careful about the fact that, if a user no longer has any unread notifications on a wiki, that wiki will stop showing up in notwikis=* (and vice versa, you might discover a new wiki later when the user first has an unread notification there). Relatedly, I'll point out that the updates you're looking for aren't only new notifications, but also existing notifications being marked as read (or as unread!) on another platform.

Could we populate the “123 messages from 6 projects” label from the API (particularly if this is paged)?

Yes you can, using the notcrosswikisummary parameter I described above. The fake notification that is returned by this (with id=-1 and type=foreign) has a count property with the number of unread foreign notifications, and a sources property with a list of wikis where those notifications live.

Screenshot from 2021-07-22 11-47-51.png (984×1 px, 154 KB)

Passing count as one of the values of notprop (which it looks like you're already doing) also gets you the total number of notifications, in the count and rawcount properties. (rawcount is a number, count is that number formatted as string in the user's language, which may use different characters to represent it; try &uselang=bn for an example of that.) You could use this if you want the count that you're showing to be consistent with the notification count that the user sees on the web.

Can we get the list of projects a user is “in” for Echo, or do we just know about them as they arrive locally?


As I said before, Echo itself only really tells you about wikis where the user has unread notifications at the time of your request. There is another API for finding out on which wikis the user has an account, so you could use that instead, but be warned that that list can be very long, since it'll be just about every wiki the user has ever visited, even if they've never done anything there. My volunteer account exists on 279 wikis, your work account exists on 320, and one of our more prolific coworkers is on 570. There are 976 wikis in total at present (some of which are private or closed, so the theoretical maximum that you could get back from this API is probably about 900-950). That is all to say that making API requests to all of a user's wikis separately might not be feasible. With the notwikis parameter you can batch up to 50 wikis at a time, so that might work if you're willing to make a half dozen requests or so (plus more to paginate it out when there are a lot of notifications, but you were already planning that I think). So instead you may want to have the user configure this list themselves, or use the top ~50 wikis where they have the most edits, or add a wiki to the list when you discover an unread notification on it through the cross-wiki feature, or some combination of those things, to keep the list of wikis you're checking to a more manageable size.

The answer to question 6 also is a little iffy since we are limited to 50 (or 500 can be arranged I'm sure) notifications to mark as unread or read. We can try splitting up the calls so maybe no design change will need to be taken, stay tuned.

The "500 for certain clients" thing is based on user rights (tied to the user account), so with the current API infrastructure we can't privilege the apps to get higher limits. (The feature was meant for bots that have their own account.) Splitting up the calls is what I would recommend.

I also wanted to point out that the Notifications API differentiates between the notifications area being "seen" vs "read".

This whole paragraph is an excellent description of how the "seen" feature works and how it differs from "read", you got it exactly right. When we worked on this, our team had trouble wrapping our heads around this and everyone was always confused when reasoning about it, but you figured out exactly how it works and wrote a better explanation of it than I did at the time. That's some impressive reverse engineering.

@Catrope wow, thank you so much for all of this! It explains everything and corrects a lot of assumptions I was making. I had the subdomain perspective and the way it affects things all wrong but I think now I get it.

If you're looking to filter by notification type, it wouldn't be that hard to add that to the Echo API.

That actually has been scrapped in our latest design mocks, but it's good to know it may not be too bad to have API support if that makes it back in for a v2.

I'm not sure if this helps to explain the behavior you saw, but notwikis=* means "all wikis where I have unread notifications, plus the local wiki".

Yes, that's why I was seeing that behavior, I only had unread notifications on EN so the other projects with read notifications were filtered out. If I pass in notwikis=enwiki|mediawikiwiki, I do see read and unread from both projects, but continue is null if I'm calling from mediawiki.org because I have very few mediawiki notifications, even though there are more pages of EN notifications. That's the perspective part that threw me off - I wouldn't be able to populate read and unread data from multiple projects in a single call, I think I'll need to break them up into en.wikipedia.org?notwikis=enwiki and mediawiki.org?notwikis=mediawikiwiki during the import phase for the continue IDs to work out.

So instead you may want to have the user configure this list themselves, or use the top ~50 wikis where they have the most edits, or add a wiki to the list when you discover an unread notification on it through the cross-wiki feature, or some combination of those things, to keep the list of wikis you're checking to a more manageable size.

Yep, we have a list of app languages that the user already has set that we can lean on, and are also planning on having a filter by project feature for these notifications. Adding the projects returned in notwikisummary is a great idea too, thanks for the heads up about that.

This whole paragraph is an excellent description of how the "seen" feature works and how it differs from "read", you got it exactly right.

Thanks! The leap from seen value to badge color was guesswork on my part, so it's good to know sometimes I get it right! 😆

Yes, that's why I was seeing that behavior, I only had unread notifications on EN so the other projects with read notifications were filtered out. If I pass in notwikis=enwiki|mediawikiwiki, I do see read and unread from both projects, but continue is null if I'm calling from mediawiki.org because I have very few mediawiki notifications, even though there are more pages of EN notifications. That's the perspective part that threw me off - I wouldn't be able to populate read and unread data from multiple projects in a single call, I think I'll need to break them up into en.wikipedia.org?notwikis=enwiki and mediawiki.org?notwikis=mediawikiwiki during the import phase for the continue IDs to work out.

You've found a bug (two bugs, really) in the limit/continue handling here that I hadn't realized existed. I don't think anything in the UI uses multiple notwikis values, which might be why we never noticed this. These are almost certainly my fault, because I wrote this code in 2016 and it hasn't changed much since.

  1. notlimit limits the number of results per wiki, not the total number of results returned. My account has 5 notifications on hewiki and 42 notifications on enwiki, so you'd expect notwikis=hewiki|enwiki&notlimit=20 to return 20 results (maybe 5 from hewiki and 15 from enwiki, or some other mix depending on how old they are). But instead, it returns 25 results (5 from hewiki and 20 from enwiki) because the limit is applied to each wiki separately. This is probably not a big problem, but it is surprising that you could set notlimit=20 and potentially get 100+ results if you listed 5+ wikis (either explicitly, or implicitly with notwikis=*)
  2. The returned continue value is based entirely on local notifications, and doesn't take foreign notifications into account at all. This means that, in the previous example (notwikis=enwiki|hewiki&notlimit=20), I get a continue value back if I send the request to en.wikipedia.org (because there are more than 20 notifications there) but if I send the same request to he.wikipedia.org I get continue: null, because I've gotten all hewiki notifications, even though there are more enwiki notifications. But the notcontinue parameter does affect the foreign notifications returned in the next request, which means that if you do get a continue value, using it will either skip or duplicate foreign notifications in your pagination.

In brief, paginating through notifications from foreign wikis is completely broken. I think #2 is fixable (by cleverly proxying through the continue value from the foreign API request that we make internally), but it would require some work and then careful testing.

@Tsevener I'm just catching-up on the conversations happening here in the comments (big thanks to Roan for all of your help!)

I'm curious if you think that it'd be better/easier for us to scrap the by-project inboxes for v1?

@cmadeo I think there's a couple of directions we could go in -

  1. I think the easiest would be displaying notification center on a per-wiki basis, since the local wiki pagination works in the API. So that would be keeping the project inbox screen, but treating it as single-select that applies to the whole in app notification center. We couldn't treat projects as an inbox filter that could be removed to show more notifications in a list combined with notifications from other projects.
  2. A second option to try would be for us to fetch notifications from all the different wikis separately under-the-hood, and display them combined in a single list. The filter will act as a local filter only on data we have already fetched and persisted (which I was already leaning towards). This method isn't totally uncharted territory - we do something similar for the Explore feed because all of that data is fetched from their respective wikis and combined rather than a single API call. We could keep our design with this method.

Regardless of which option we choose, I am starting to feel like we should scrap the bulk import. Scrapping the bulk import would just mean we would want to change the counts text at the bottom from "[n] notifications in [n] projects" to "[n] unread notifications in [n] projects". The unread number across projects is something that can be easily fetched in a single call from the server.

Thanks for this update and clarification @Tsevener! I like the second approach (seems closest to the current design proposal 😉), but happy to make changes as things come up.

Also I'll update the mocks to include the 'unread' element in the thread. Thanks!

Hi @Catrope, thanks for the heads up about the bugs! We're still talking it over on our side how to adjust.

I'm not sure if you're the right person to ask on this (feel free to tag someone else if not), but do you think if we broke it up into separate calls (en.wikipedia.org?notwikis=enwiki, es.wikipedia.org?notwikis=eswiki, etc.) for each app language the user has set, the server would be able to handle the load? Particularly if we decide to do a bulk import, so we'd be fetching N number of projects, paged to the beginning of notification time in one go. Android bulk imports notwikis=*, so this bulk import method would be a different beast than theirs since we'd like both read and unread notifications from all app language wikis they have set up.

Hi @Catrope, thanks for the heads up about the bugs! We're still talking it over on our side how to adjust.

I'm not sure if you're the right person to ask on this (feel free to tag someone else if not), but do you think if we broke it up into separate calls (en.wikipedia.org?notwikis=enwiki, es.wikipedia.org?notwikis=eswiki, etc.) for each app language the user has set, the server would be able to handle the load? Particularly if we decide to do a bulk import, so we'd be fetching N number of projects, paged to the beginning of notification time in one go.

I'm not totally the right person to ask, but I'm also not sure that that right person exists :) . I think what you propose will be fine, especially since I'm guessing that the number of app languages users typically set is low (I imagine it's rarely higher than 10). A bulk import would also happen only once per user. For updating the app with subsequent notifications, how often would you make API requests, and would those be for all wikis or only the selected one?

(Also note that the notwikis parameter is optional and defaults to the current wiki, so in your example you could omit it.)

@Catrope yeah for subsequent notifications, we'd fetch the first page for each wiki and only page further if it seems like there's more than one page of new stuff. It would be ideal to do this whenever they entered the app notification center, foregrounded on the app notification center, received a push while in the app notification center, or pulled to refresh in the app notification center.

To try to reduce the number of calls we could add a minimum time passed since last refresh rule for the first two. The user can also have a local filter set for projects, so we can only refresh the ones they have set to display. And when we receive a push, if we can determine which project the push is from, we can only refresh that one.

I also feel like we should trash the local data and re-do the bulk import if the user logs out and logs back in.

(Also note that the notwikis parameter is optional and defaults to the current wiki, so in your example you could omit it.)

Ah I missed that, thanks!

OK, that sounds fine. To reduce concerns about server load, you could also make these requests in series rather than in parallel, if you think that won't degrade the user experience.

Thanks for looking into this. If I am understanding this correctly, from the Android perspective we don't need to make any changes at this time.