It's 6 hours later and I still have the original issue on enwiki. Here's something interesting: I am also getting a ghost notification on the metawiki side. My tray says "1", but everything in my tray is marked as read.

Maybe someone with database access can run the following SQL query (or similar) and post the results here so we can see what the ghost notification looks like in the database? My guess is it's malformed.

USE metawiki;
SELECT *
FROM echo_notification
JOIN user ON user_id = notification_user
WHERE user_name  = 'Novem Linguae'
AND notification_read_timestamp IS NULL

Maybe someone with database access can run the following SQL query (or similar) and post the results here so we can see what the ghost notification looks like in the database? My guess is it's malformed.

mysql:research@dbstore1008.eqiad.wmnet [metawiki]> SELECT *
    -> FROM echo_notification
    -> JOIN user ON user_id = notification_user
    -> WHERE user_name  = 'Novem Linguae'
    -> AND notification_read_timestamp IS NULL;
Empty set (0.001 sec)

(I verified that the query is correct, it returns some rows for myself when I have an unread notification)

The numbers shown on the badges are cached: https://gerrit.wikimedia.org/g/mediawiki/extensions/Echo/+/7845b5fab1ff256e83bb2bba2b944164dc84460d/includes/NotifUser.php#565

The un-dismissible notification went away two days later when I received an additional notification on meta.

My attempts before that to make it go away, such as marking it as read and unread on both enwiki and meta, failed.

Hypothesis: maybe the cache and the database are getting out of sync?

Only happens cross-wiki.

Thanks for reporting and looking into this, @Novem_Linguae!
@Etonkovidova do you think this is the same underlying issue as T368643: Not possible to mark cross-wiki notifications as marked in bulk?

In T369451#9978140, @KStoller-WMF wrote:

Thanks for reporting and looking into this, @Novem_Linguae!
@Etonkovidova do you think this is the same underlying issue as T368643: Not possible to mark cross-wiki notifications as marked in bulk?

T368643: Not possible to mark cross-wiki notifications as marked in bulk seems to be a different issue and it's easily reproducible. This issue seems more elusive.So far, I was not able to reproduce it. When I was checkin, the status change in cross-wiki notifications got synchronized practically instantaneously. However, I remember that at some point there were intermittent failures for such synchronization from certain wiki, and metawiki was one of them.

T368643: Not possible to mark cross-wiki notifications as marked in bulk has a merged change that rolls out next Thursday. Given how similar these tasks sound let's check after that if this problem here still occurs.