I still see this a lot. (Always with OAuth admin notifications, but that might just be because that's the type of notification I usually get from meta.)

An example of the action API request on the non-Meta wiki (mediawiki.org, in this case):

https://www.mediawiki.org/w/api.php?action=query&format=json&formatversion=2&meta=notifications&notsections=alert&notformat=model&notlimit=25&notprop=list%7Ccount%7CseenTime&uselang=en&notwikis=metawiki&notfilter=!read&notbundle=true&_=1689094065889

{
    "batchcomplete": true,
    "query": {
        "notifications": {
            "list": [],
            "rawcount": 0,
            "count": "0"
        }
    }
}

An example of the request on Meta:

https://meta.wikimedia.org/w/api.php?action=query&format=json&meta=notifications&notsections=alert&notgroupbysection=1&notmessageunreadfirst=1&notlimit=25&notprop=count&uselang=en&notcrosswikisummary=1&_=1689094179431

{
    "batchcomplete": "",
    "query": {
        "notifications": {
            "alert": {
                "rawcount": 1,
                "count": "1"
            },
            "rawcount": 1,
            "count": "1"
        }
    }
}

(The unread notification in this case was ID 4722181, type oauth-app-propose.)

If I send a more exact equivalent of the first API request to Meta:

https://meta.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&meta=notifications&notsections=alert&notformat=model&notlimit=25&notprop=list%7Ccount%7CseenTime&uselang=en&notfilter=!read&notbundle=true&_=1689094065889

I get the notification as expected.

I'm also seeing this bug today, but for a Thanks notification at Meta-wiki. However it seems to affect all types, and it's only not working for me at Mediawiki-wiki.
I.e. If I mark a "mention" or "reply" on Frwiki and Enwiki as unread, then the cross-wiki notifications at Mediawiki-wiki still fail to load (screenshot), but I can see them fine at Meta-wiki or Swwiki.
That doesn't really help narrow it down, but does eliminate the possibility of it just being from "Notifications from optional extensions" (OAuth and Thanks).

I am not sure how this related to T342201, but it seems obviously related.

In T223413#9240943, @matmarex wrote:

I am not sure how this related to T342201, but it seems obviously related.

That issue seems to be cause by mcrouter failure and is very random, while this one is deterministic. At most, both are related to T347781: MWHttpRequest should not route read requests to the primary DC, but even that's unlikely given $wgLocalHTTPProxy was introduced in T288848: Make HTTP calls work within mediawiki on kubernetes in 2021 and this bug is way older than that (or cross-DC MediaWiki requests). I guess it could have two different causes...

I'm also seeing this - notifications rarely if ever load when they're from a wiki that's different to the one I'm on.

I wanted to create a new task, but I think that this one is actually the same. If this isn't the same, let me know, and I'll open a new one, I didn't want to create a duplicate.

I just opened mediawiki.org, and there are notifications from Serbian Wikipedia. When I click on alerts and then expand, it doesn't display anything on mediawiki.org, like it's supposed to.
When I go on Serbian Wikipedia, it works normally.

Having the same issue and it's occurring on the meta-wiki side as well.

I can't imagine why calling the primary datacenter would be a problem in this case, unless there is a logical race condition or, worse, we completely rely on data being in memcached and/or any other datastore that's not replicated.

I don't think T347781 is the right way to go, honestly. I'd rather not duplicate the logic used at the edge inside of mediawiki itself if at all possible.

So let me rewind a bit and ask: why calling the primary datacenter would be causing an issue? What data pertaining to echo notifications can't be found in the primary dc?

We need to dig a bit more to ensure what triggers this:

• Does this happen only when people are normally directed to the secundary datacenter? Or is this an issue even when people hit the current primary?
• What url gets requested by MediaWiki? What changes in the request in the two cases? Just the destination datacenter or something else? For instance, we're going via the service mesh when using the local proxy; it would be interesting to try to see what happens if we instead go to the edge caches via the webproxy instead.

I suspect the fix I made for T342201 actually might have solved this issue as well. Not sure how to verify it though.

@matmarex do you have a way to verify if the bug still presents itself? It's slightly hard for me as I mostly edit mediawiki.org or wikitech.

In T223413#9308043, @Joe wrote:

I can't imagine why calling the primary datacenter would be a problem in this case, unless there is a logical race condition or, worse, we completely rely on data being in memcached and/or any other datastore that's not replicated.

This issue doesn't look like a memcached failure to me. T342201 & co are related to the CentralAuth token store, one of the few components which use memcached not as a cache but as a way to exchange information (in that case, a proof-of-authenticity token) between servers. If the read fails, the entire interaction fails.

That's not the case here - the Echo request doesn't fail, it just has incorrect results. Unless there is some very broken error handling somewhere in Echo (possible I guess), I don't see how it could be cause by an object cache lookup failure. And as I said, this error seems completely deterministic and independent of which DC you are being routed to.

I don't think T347781 is the right way to go, honestly. I'd rather not duplicate the logic used at the edge inside of mediawiki itself if at all possible.

T347781 is a bug in its own right and should be fixed, regardless of whether it causes this one (I don't think so - this issue long precedes the existence of $wgLocalHTTPProxy). Duplicating logic is not great. Not applying the logic to operations it should apply to is definitely worse.

In T223413#9309890, @Tgr wrote:

That's not the case here - the Echo request doesn't fail, it just has incorrect results. Unless there is some very broken error handling somewhere in Echo (possible I guess), I don't see how it could be cause by an object cache lookup failure. And as I said, this error seems completely deterministic and independent of which DC you are being routed to.

I think the possible reason is that in the case of kubernetes, you'd get the information from two different clusters of memcache between requests, while you'd consistently go to the primary-dc one on bare metal, due to the misconfiguration I described. Basically I'm saying that this problem should only have been present if your request was sent to the secondary DC (which is currently true for anyone in say europe or africa).

It's perfectly possible there are multiple bugs that can cause cross-wiki notifications to fail, but I can completely see failure scenarios in which we get the different results due to caches being written inconsistently between datacenters.

For me, I am in USA, and the bug for MediaWiki.org notifications trying to display enwiki notifications went away about a month ago. Was seeing it consistently before, and haven't seen it since. Hope this info helps.

In T223413#9313665, @Novem_Linguae wrote:

For me, I am in USA, and the bug for MediaWiki.org notifications trying to display enwiki notifications went away about a month ago. Was seeing it consistently before, and haven't seen it since. Hope this info helps.

yes that would fit with the timeline of the switchover (which happens on the week of the equinox) and would further confirm my suspicions that my asinine error was the main cause.

I still don't get how a fix to the networking of k8s could have fixed a bug that precedes k8s, but I can confirm that the bug is not happening anymore.

(Except when notifications get removed, e.g. because the corresponding page gets deleted, so by the time you click to expand the "More alerts from another wiki" line, there is nothing to show. That still happens sometimes, but is unrelated and more of an UX issue than a problem with Echo's notification loading logic.)