No on-site Notifications received for certain type of actions
Closed, ResolvedPublic

Description

NOTE: if you are not receiving notifications, editing your Notifications preferences seems to do the trick (i.e. make a change, save, revert the change, save).

Notifications seems to be broken.

On Structured Discussions

  • no notification received on site for:
    • new replies on a topic I watch
    • edits made on a topic I watch

Notifications by email have been received, so Watchlists inputs.

Mention on a talk page

Crosswiki notifications
@Etonkovidova checked hewiki- reply and mention are received from testwiki, so cross wiki notifications are ok

Thanks for X edits
Works.

[ongoing investigation]


Initial report
Hello. It's about 12 hours that Echo does not send Flow notifications. For example, I know that there were edits in w:he:Topic:Tzpocwgmtnynbibm only from the watchlist, they do not and did not appear in Special:Notifications, neither new topic created, and after I added to the watchlist nor the answer post. Could you check this, please? Thank you.
Update 1: It's not just Flow, Echo itself does not work, users complain that notice user alerts do not work.
Update 2: A little part of notifications come, but most of them don't.

There are a very large number of changes, so older changes are hidden. Show Older Changes
Cirdan added a subscriber: Cirdan.Oct 11 2017, 7:39 PM

The problem appears on de-wp as well, it was reported by a user on our central discussion page today.

For example, this edit did not result in a ping notification for me.

He7d3r added a subscriber: He7d3r.Oct 12 2017, 12:25 AM

Users are reporting the same kind of issue on ptwiki. For example, I didn't get a notification from this edit:
https://pt.wikipedia.org/w/index.php?diff=50111696
[copied from T174873#3678440]

Re-checked notifications on hewiki (1.31.0-wmf.3) - sent notifications with enabled Failure/Success Mention in the following format from the Flow and wikitext pages:

[[User:username]] ~~~~
{{Ping|username}} ~~~~

The mention notifications were sent (the success/failure notificaitons were sent to the sender) and received by the recipients. I will be checking for other wikis tomorrow when 1.31.0-wmf.3 will deployed.

In the past 48 hours:

  • Multiple users report on nlwikinews that pings to user names do not work.
  • On Outreach wiki my user rights changed, no notification received.

In both cases with similar cases earlier notifications were received, notifications still enabled in preferences.

Two days ago I also noticed that I did not receive any OTRS notification by email either, while normally I get one with a new message. (Maybe related, maybe not.)

Tested in metawiki - it seems that mentions in format {{ping|username}} without a signature are not sent from wikitext pages. Success/failure options for Mention notifications won't register {{ping|username}} without a signature as a failure to sent mention.

Stryn added a comment.Oct 12 2017, 3:53 PM

Tested in metawiki - it seems that mentions in format {{ping|username}} without a signature are not sent from wikitext pages. Success/failure options for Mention notifications won't register {{ping|username}} without a signature as a failure to sent mention.

But they always have required a signature to work.

On ca.wiki it is reported that it works saving user preferences with some change. I tried changing notification to email, then changing back to "web" and now it is working for me, both notification for mention and structured discussion.

I meant - if the signature will be directly placed after ping mention, e.g. {{ping|username}} ~~~~, but not like

::{{ping|User1}} Testing mentions in ping format with text after ping and mentioning  another user in a different format [[User:User2]] signature comes here ~~~~

When the above example will be used - no notifications will be send, and no failure/success mention notifications will be registered either. Will ask @Catrope to look in the logs ; yesterday there were some echo parser errors - not sure how relevant they are to the issue.

Johan moved this task from To Triage to Archive on the User-notice board.Oct 12 2017, 5:54 PM
Johan moved this task from Archive to In current Tech/News draft on the User-notice board.
IKhitron added a comment.EditedOct 12 2017, 6:52 PM

I just published a new MediaWiki-extensions-Newsletter issue. Some of notifications in email arived, some didn't.

I did this again, and another partition of the subscribers did not get the message.

Etonkovidova added a comment.EditedOct 12 2017, 8:08 PM

Checking enwiki (wnf.3) - some mentions do come (cannot see any pattern); the success mention notification "Your mention to ... was sent" is displayed, but the recipient does not receives the mention notification - not within 10 min and counting.

Note: Also, after enabling "Failed mention/Successful mention" options I received 6-month old notifications (marked as unread).

QuimGil added a subscriber: QuimGil.EditedOct 13 2017, 11:29 AM

I just published a new MediaWiki-extensions-Newsletter issue. Some of notifications in email arived, some didn't.

Confirmed. Notifications about new Newsletter issues are not arriving.

Specifically, this issue posted today hasn't generated the expected notifications as far as I can see (I am subscribed there with my work and personal accounts):

https://www.mediawiki.org/wiki/Newsletter:Newsletter%C2%B2

12:08, 13 October 2017 Qgil-WMF (talk | contribs | block) published a new issue of Newsletter² newsletter at Newsletter:Tech Showcase (New newsletter: Tech Showcase)

On ca.wiki it is reported that it works saving user preferences with some change. I tried changing notification to email, then changing back to "web" and now it is working for me, both notification for mention and structured discussion.

I think I can replicate this. After changing notifications preferences, now notifications seem to work both in my personal account in Catalan Wikipedia and my work account in MediaWiki.org.

Elitre added a subscriber: Elitre.Oct 13 2017, 2:40 PM

Confirmed. Notifications about new Newsletter issues are not arriving.

After editing my Notifications preferences, a new Newsletter notification is shown in my notifications correctly.

Same here in german wiktionary. See discussion: https://de.wiktionary.org/wiki/Wiktionary:Teestube#Pingen_scheint_.28wieder_einmal.29_nicht_zu_funktionieren
After changing my preferences it works for me, but nevertheless, there should be a general solution for all users, without the need for changing their preferences themself.

(I think the workaround, if any, could be explained in the description more prominently? Haven't tried to apply it yet.)

DatGuy added a subscriber: DatGuy.Oct 14 2017, 1:32 PM

There are currently complaints on the English Wikipedia that mentions/pings are also not sending emails, not just not giving on-site notifications.

I can also confirm that I have not been receiving pings/mentions on en.wiki.

I haven't been receiving any pings/mentions since the beginning of this week. I just noticed that I did not receive notification for incoming Wikipedia e-mail as well. When can this be fixed?

JJMC89 added a subscriber: JJMC89.Oct 14 2017, 8:01 PM

I'm investigating this now, but it seems like my accounts are not affected. From an incognito window, I pinged my work account and my personal account and I got a web notification both times. I then enabled emails for mentions on my personal account, pinged it again, and that worked too.

I'm now going to try pinging @GeneralizationsAreBad and @alex.shih from the enwiki sandbox, because they both say that it isn't working for them and their reports are quite recent. If you get notifications about those pings, I apologize for the noise.

QuimGil updated the task description. (Show Details)Oct 14 2017, 10:29 PM
Catrope added a comment.EditedOct 14 2017, 10:48 PM

I found a problem that may or may not be related:

mysql:research@s3-analytics-slave [enwiki]> select * from user_properties where up_user=REDACTED and up_property='echo-notifications-blacklist';
+----------+------------------------------+----------+
| up_user  | up_property                  | up_value |
+----------+------------------------------+----------+
| REDACTED | echo-notifications-blacklist | 0        |
+----------+------------------------------+----------+
1 row in set (0.00 sec)
> var_dump($user->getOption('echo-notifications-blacklist'));
array(1) {
  [0]=>
  int(0)
}

On enwiki, I found 1376 users who have their blacklist pref set to 0, and also one whose blacklist preference is 0\n0 (two zeroes separated by a newline). This strongly suggests that updatePerUserBlacklist.php had a bug. T177437 shows logs of at least one abortive attempt at running the maintenance script and suggests there were more (it says it failed "again").

I suspect (but haven't yet proven) that this causes the issue. The script was run on Oct 9, and this bug was reported on Oct 10. Even if this script didn't cause the issue, it zeroed out the blacklists of 1377 users on enwiki and probably hundreds of users on other wikis as well (332 on dewiki for example), which is a problem of its own.

Query to find broken blacklists:

mysql:research@s3-analytics-slave [dewiki]> select count(*) from user_properties where up_property='echo-notifications-blacklist' and (up_value='0' or up_value like '0\n%' or up_value like '%\n0' or up_value like '%\n0\n%');
+----------+
| count(*) |
+----------+
|      332 |
+----------+
1 row in set (5.32 sec)

Local testing confirms that a blacklist preference set to 0 suppresses all notifications coming from non-anonymous users (which is kind of surprising, you'd expect the opposite, since the user ID of an anonymous user is 0).

Local testing confirms that a blacklist preference set to 0 suppresses all notifications coming from non-anonymous users (which is kind of surprising, you'd expect the opposite, since the user ID of an anonymous user is 0).

I have figured out why. It's a combination of extremely counter-intuitive behavior in both CentralAuth and PHP.

Suppose a user's blacklist preference is set to "0", then $user->getOption('echo-notifications-blacklist') will return [0] (an array with one element, whose value is the integer 0). NotificationController::isBlacklistedByUser will then create a $blacklist for this user (an instance of EchoContainmentSet) and call $blacklist->addFromUserOption( 'echo-notifications-blacklist' );.

addFromUserOption expects an array of central user IDs, and resolves these to user names: $names = $lookup->lookupCentralIds( array_flip( $preference ), $this->recipient );. This returns an array where the keys are central user IDs and the values are user names. But what it's technically doing is replacing the values with user names in the array it's given, so if the array you give it is [0 => 0], it will fail to look up a user name for ID 0, and return [0 => 0] (AKA [0]). As a further example, $lookup->lookupCentralIds(array_flip([57,1,0]) returns [57 => 0, 1 => 'Admin', 0 => 2] on my test install.

EchoContainmentSet uncritically accepts the number 0 as a user name and puts it in an array. Then when $blacklist->contains( 'Catrope' ) is called, it does array_search( 'Catrope', [ 0 ] ) internally. Surprisingly, this returns 0 (found at index 0) rather than false (not found), because under the weak comparison rules, 'Catrope' == 0 is true. array_search( 'Catrope', [ 0 ], true ) does return false, because the third parameter forces a strict comparison.

I'm going to prepare patches to have lookupCentralIds set values to null when an ID can't be resolved to a user name,; to filter out these null values when inserting into EchoContainmentSet; and to use strict comparisons in EchoContainmentSet::contains. The latter alone is enough to fix the bug, so I'll do that one first.

Change 384299 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/Echo@master] ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384299

Change 384300 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/Echo@master] ContainmentSet: Work around unhelpful behavior of lookupCentralIds()

https://gerrit.wikimedia.org/r/384300

I'm going to prepare patches to have lookupCentralIds set values to null when an ID can't be resolved to a user name,

I ended up not doing this, because this (unhelpful) behavior is well documented. Instead, I changed the code in ContainmentSet to set all array values to null in the array fed to lookupCentralIds, then filter out nulls in the result.

Change 384299 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/Echo@master] ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384299

We should SWAT this patch on Monday to make the bug stop happening. @dbarratt could you please review it ASAP?

In the meantime, affected users can make a change (any change) to their preferences. Saving one's preferences clears out the zeroes that are causing this bug.

Change 384299 merged by jenkins-bot:
[mediawiki/extensions/Echo@master] ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384299

wow. @Catrope thanks for finding / fixing that! Does it need to be scheduled for SWAT deploy?

T177437 shows logs of at least one abortive attempt at running the maintenance script and suggests there were more (it says it failed "again").

I suspect (but haven't yet proven) that this causes the issue. The script was run on Oct 9, and this bug was reported on Oct 10. Even if this script didn't cause the issue, it zeroed out the blacklists of 1377 users on enwiki and probably hundreds of users on other wikis as well (332 on dewiki for example), which is a problem of its own.

Yeah that sounds like the problem, but I can dig in and confirm. I didn't attempt to run the script on entries that were already valid ids. I'm assuming that would cause the behavior you've seen. :(

wow. @Catrope thanks for finding / fixing that! Does it need to be scheduled for SWAT deploy?

If possible ASAP, please!

wow. @Catrope thanks for finding / fixing that! Does it need to be scheduled for SWAT deploy?

If possible ASAP, please!

Scheduled for today's morning SWAT, I'll cherry-pick the patch on the wmf branch.

Change 384536 had a related patch set uploaded (by Dbarratt; owner: Catrope):
[mediawiki/extensions/Echo@master] ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384536

Change 384536 abandoned by Dbarratt:
ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384536

Change 384537 had a related patch set uploaded (by Dbarratt; owner: Catrope):
[mediawiki/extensions/Echo@wmf/1.31.0-wmf.3] ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384537

Change 384537 had a related patch set uploaded (by Dbarratt; owner: Catrope):
[mediawiki/extensions/Echo@wmf/1.31.0-wmf.3] ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384537

This is against the correct branch.

SPoore added a subscriber: SPoore.Oct 16 2017, 3:35 PM

Change 384537 merged by jenkins-bot:
[mediawiki/extensions/Echo@wmf/1.31.0-wmf.3] ContainmentSet: Use strict comparison for array_search()

https://gerrit.wikimedia.org/r/384537

dbarratt moved this task from Ready to In progress on the Anti-Harassment (AHT Sprint 7) board.
dbarratt added a subscriber: dmaza.

Ah, pages in the User Talk pages are excluded from Echo Notification mute. I also needed the latest master of MediaWiki and that resolved my issue(s). Good to deploy now! I'll put it back on the SWAT schedule.

Mentioned in SAL (#wikimedia-operations) [2017-10-16T23:25:15Z] <thcipriani@tin> Synchronized php-1.31.0-wmf.3/extensions/Echo/includes/ContainmentSet.php: SWAT: [[gerrit:384537|ContainmentSet: Use strict comparison for array_search()]] T177825 (duration: 01m 45s)

dbarratt closed this task as Resolved.
dbarratt claimed this task.

Apparently... in T178313: Recover Echo Notification Blacklist from Backup we discovered that we did not lose anyone's echo notification blacklist.

Does anyone who had this problem know of any steps they did that may have caused their blacklist to be 0 in the database? I cannot reproduce a scenario in which this happens.