No page link notifications since around end of June
Closed, ResolvedPublic

Description

We are still collecting information, but several users of the German-language Wikipedia are reporting that they have not received any Echo notification that one of their articles was linked since end of June. (permalink to current state/live link (will be archived))

The two users who have confirmed they stopped receiving notifications are Hoefler50 and Lómelinde. We are still trying to find out if this affects all users or just a certain group.

Since the change to global preferences falls into this timeframe, it might be that this Echo problem is related.

Cirdan created this task.Jul 21 2018, 7:23 AM
Restricted Application added a project: Growth-Team. · View Herald TranscriptJul 21 2018, 7:23 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Cirdan updated the task description. (Show Details)Jul 21 2018, 7:26 AM

Examples of links added for which no notification was received by user Lómelinde:

Cirdan renamed this task from No page link notifications on dewiki since around end of June (investigating) to No page link notifications on dewiki since around end of June.Jul 21 2018, 11:29 AM

Another affected user is Diwas, who is the creator of the page "China" which according to them is linked to almost daily.

He last received a notification for this edit on June 28th.

This might or might not be related to T200823: Site stats not updating when links added to pages, which is also about links added to pages not triggering events.

Reedy closed this task as Resolved.Aug 3 2018, 2:20 PM
Reedy claimed this task.
Reedy added a subscriber: Reedy.

Underlying bug fixed. Stats updated

reedy@deploy1001:/srv/mediawiki-staging$ mwscript updateArticleCount.php --wiki=dewiki --update
Counting articles...[Fri Aug  3 14:19:26 2018] [hphp] [19768:7f6c132423c0:0:000001] [] SlowTimer [51170ms] at runtime/ext_mysql: slow query: SELECT /* SiteStatsInit::articles www-data@deploy... */  COUNT(DISTINCT page_id)  FROM `page`,`pagelinks`    WHERE page_namespace = '0' AND page_is_redirect = '0' AND (pl_from=page_id)  LIMIT 1  
found 2205826.
Updating site statistics table... done.
Cirdan added a comment.Aug 4 2018, 9:06 AM

@Reedy How certain are you that this fixes the problem? When should users expect to see notifications again?

Reedy added a comment.Aug 4 2018, 10:47 AM

Uhh. I thought this was a different task related to article counting :(

Dunno if this is actually fixed then...

Cirdan reopened this task as Open.Aug 4 2018, 11:30 AM

OK, let's keep this open and see what happens. We're still not sure if everybody is affected, but it's likely that this task here is also related to links to pages being added is not detected correctly.

Cirdan removed Reedy as the assignee of this task.Aug 4 2018, 6:50 PM
Catrope closed this task as Resolved.Aug 20 2018, 11:04 PM
Catrope claimed this task.
Catrope added a subscriber: Catrope.

Closing this in the absence of any new reports in the 2+ weeks since @Reedy claimed to have fixed it. Please reopen if you find new occurrences.

Cirdan reopened this task as Open.Aug 21 2018, 9:19 AM

Hi @Catrope, there are still no page link notifications on dewiki. None of the users who complained has received any notifcations since the end of June, and as far as we can tell without surveying all users, nobody receives any page link notifications.

Reedy mistakenly took my comment that this bug might be related to the other one as meaning that there definitely is a connection and therefore closed this task without verifying that their update fixes this problem (which it doesn't).

You're right that something strange is happening. I can see a bunch of recent echo_event rows with event_type='page-linked', but none of them are referred to by rows in echo_notification. The most recent rows in echo_notification that refer to a page-linked event are from 2018-06-28, with the very last one being at 20:46:39 UTC. This lines up perfectly with:

20:46 marxarelli: Rolling 1.32.0-wmf.10 to group2 following fix and successful re-deploy to group1

So it looks like something happened in wmf.10 that broke this and hasn't been fixed since.

Thanks for investigating! Any idea what could have caused this?

From looking through the task tagged with MW-1.32-release-notes_(WMF-deploy-2018-06-26_(1.32.0-wmf.10)) I don't see anything related to Echo or notifications. There seem to have been some changes to page saving which caused issues (T198483) and there is (T200823) which is related to detecting link updates, although from what you wrote, the fact that a link was added is recorded correctly.

Looks like page-linked notifications are broken everywhere. They also haven't been generated on English Wikipedia since June 28th, and I can even reproduce this exact same behavior (echo_event row but no echo_notification row) locally. That should make it easier to debug at least.

Somehow, the triggeringUser parameter to LinksUpdate/RefreshLinksJob is no longer being passed in. This means the page-linked event is saved to the database with the agent set to null, rather than the user that made the edit. EchoUserLocator::locateArticleCreator() then sees there is no agent and short-circuits, returning an empty list of recipients rather than looking up the page creator.

I can confirm this broke in MW core somewhere between wmf.8 and wmf.10, bisecting now.

Jdforrester-WMF renamed this task from No page link notifications on dewiki since around end of June to No page link notifications since around end of June.Aug 24 2018, 10:11 PM

Change 455259 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/Echo@master] Fix page-linked notifications

https://gerrit.wikimedia.org/r/455259

Change 455401 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@master] Restore passing of triggering user in edit updates

https://gerrit.wikimedia.org/r/455401

daniel added a subscriber: daniel.Aug 27 2018, 1:42 PM

Looks like I screwed this up. Thanks for finding the issue!

Change 455401 merged by jenkins-bot:
[mediawiki/core@master] Restore passing of triggering user in edit updates

https://gerrit.wikimedia.org/r/455401

Tgr closed this task as Resolved.Aug 27 2018, 2:37 PM
Tgr added a subscriber: Tgr.

Given that this wasn't noticed for weeks, it's probably not severe enough to need a backport, so closing as resolved. Please reopen if that's not the case.

Change 455259 merged by jenkins-bot:
[mediawiki/extensions/Echo@master] Fix page-linked notifications

https://gerrit.wikimedia.org/r/455259