Page MenuHomePhabricator

Fix new topic notification on MediaWiki.org
Closed, ResolvedPublic

Description

I did a test, and it's taking longer than I expected (and hasn't been delivered yet)

These wikis are:

testwiki
test2wiki
mediawikiwiki

[Update, 24 August:] Notifications for new topics are still not arriving at mediawiki.org, bumping to unbreak now. Discussed at https://www.mediawiki.org/wiki/Topic:Snfgy7p12r7jbjen

(If it's from the DB issue, the actual list is labswiki, mediawikiwiki, metawiki, officewiki).

Event Timeline

Mattflaschen-WMF raised the priority of this task from to Needs Triage.
Mattflaschen-WMF updated the task description. (Show Details)
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Mattflaschen-WMF renamed this task from Make sure new topic notification is working on wikis with job queue to Fix new topic notification on wikis with job queue.Aug 5 2015, 11:54 PM
Catrope triaged this task as High priority.Aug 5 2015, 11:55 PM
Catrope added a subscriber: Catrope.
DannyH lowered the priority of this task from High to Medium.Aug 20 2015, 8:41 PM
DannyH raised the priority of this task from Medium to High.
DannyH added a subscriber: DannyH.
Quiddity raised the priority of this task from High to Unbreak Now!.Aug 24 2015, 5:17 PM
Quiddity updated the task description. (Show Details)
Quiddity updated the task description. (Show Details)

Probably the cause (slightly redacted by changing user and event ID actual integers to just 'int'), from fluorine:

2015-08-24 20:21:38 mw1013 mediawikiwiki exception ERROR: [6ecdbbcf] /rpc/RunJobs.php?wiki=mediawikiwiki&type=EchoNotificationJob&maxtime=30&maxmem=300M   DBQueryError from line 1131 of /srv/mediawiki/php-1.26wmf19/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO `echo_target_page` (etp_user,etp_page,etp_event) VALUES (int,int,int)
Function: EchoTargetPageMapper::insert
Error: 1062 Duplicate entry 'int-int' for key 'echo_target_page_user_event' (10.64.16.27)
@db1044 [mediawikiwiki]> SHOW CREATE TABLE echo_target_page;
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table            | Create Table                                                                                                                                                                                                                                                                                                                                                                            |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| echo_target_page | CREATE TABLE `echo_target_page` (
  `etp_user` int(10) unsigned NOT NULL DEFAULT '0',
  `etp_page` int(10) unsigned NOT NULL DEFAULT '0',
  `etp_event` int(10) unsigned NOT NULL DEFAULT '0',
  UNIQUE KEY `echo_target_page_user_event` (`etp_user`,`etp_event`),
  KEY `echo_target_page_user_page_event` (`etp_user`,`etp_page`,`etp_event`)
) ENGINE=InnoDB DEFAULT CHARSET=binary |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

This is because I didn't realize that some of the Echo wikis were still using local DBs for Echo tables, so I forgot to ask for T94427: Perform schema change to echo_target_page changing from a 1 to 1 mapping between pages and user/notification to a 1 to many. to be applied there.

If that is the cause, then it is not jobqueue-specific but rather due to https://git.wikimedia.org/blob/operations%2Fmediawiki-config/20e51990e4775c1fffa0380aab5b94b2ba1c109e/wmf-config%2FInitialiseSettings.php#L14509 .

I'm not 100% sure, but I believe my original test was on MediaWiki.org and that is definitely broken now.

I've performed an emergency alter following the directions of the reopened T94427, on mediawikiwiki, officewiki, metawiki and labswiki. Please indicate if that helps on fixing the issue.

Please indicate if that helps on fixing the issue.

Thanks for your work! Maybe I'm to impatient, but I haven't received a notification on for https://www.mediawiki.org/wiki/Topic:Snny8vja9x3weask :(

Please also be aware that some jobs (including, probably, notifications) may have been lost for a short period of time due to an unrelated problem.

However, that was solved more than 1 hour ago and should not affect following jobs.

I got one apparent success 8 hours ago, but I'm still testing.

I got the two tests I did then (as a bundled, since they were to the same page).

And there's nothing in the current exception.log. Going ahead and moving this to QA.

It seems to work now (I got some notifications about new topics on mediawiki.org). Thanks for the fast fix :)

Tested in betalabs&production

  1. A new topic posted as the first new topic on a Flow page - an email notification is sent immediately
  2. Posting more new topics on the same page - email notification is sent with 4-5-hour delay. All new topics(except the first one) are bundled together, e.g.

On Talk:ET4 (http://en.wikipedia.beta.wmflabs.org/wiki/Talk:ET4) user ET1 created two topics - at 22:10 and 22:11 and
user ET2 created a new topic at 22:15. Three topics were created.

  1. Watchlist shows that Aug26-01 on Talk:ET4; 22:10 . . (+8)‎ . . ET1 (Talk | contribs | block) (←Created new topic) Aug26-02 on Talk:ET4; 22:11 . . (+8)‎ . . ET1 (Talk | contribs | block) (←Created new topic) Aug26-03 on Talk:ET4; 22:15 . . (+8)‎ . . ET2 (Talk | contribs | block) (←Created new topic)
  1. View history: http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk:ET4&action=history

22:15, 26 August 2015 . . ET2 (Talk | contribs | block) commented on "Aug26-03" (added by ET2) . . (+12)
22:11, 26 August 2015 . . ET1 (Talk | contribs | block) commented on "Aug26-02" (added by ET1) . . (+12)‎
22:10, 26 August 2015 . . ET1 (Talk | contribs | block) commented on "Aug26-01" (added by ET1) . . (+12)‎

  1. Imediately after creating a new topic tt 22:10(3:10 pm PDT) I received an email notification:

"ET1 created a new topic with the title "Aug26-01" on Talk:ET4"

  1. After exactly four hours + one minute, I received an email notification:

"3 new topics on Talk:ET4"

Problems

  • a big delay in notification sent after a new topic is posted
  • misleading info to a user in the second email(that bundles facts about topic creation): "3 new topics on Talk:ET4". A user already received a new topic notification about the first one of those three new topics.

Sadly, this is expected behavior. I was hoping to do more work on notifications, watchlists and emails, but it looks like we're not going to have time for it right now.