Page MenuHomePhabricator
Create Task
Maniphest T93609

Duplicate key errors in EchoNotificationMapper::insert
Closed, DeclinedPublicPRODUCTION ERROR
Actions

Description

On Beta Cluster:

015-03-22 21:14:26 deployment-jobrunner01 enwiki: [3ed44ff4] /rpc/RunJobs.php?wiki=enwiki&type=EchoNotificationJob&maxtime=60&maxmem=300M   DBQueryError from line 1249 of /srv/mediawiki/php-master/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO echo_notification (notification_event,notification_user,notification_timestamp,notification_read_timestamp,notification_bundle_base,notification_bundle_hash,notification_bundle_display_hash) VALUES ('165047','54','20150322191043',NULL,'1','330b53b96bc3d385195c5e8d5cbb5648','a6b08d6c31f722ac1954c4a688d2f8f3')
Function: EchoNotificationMapper::insert
Error: 1062 Duplicate entry '54-165047' for key 'user_event' (10.68.16.193)

#0 /srv/mediawiki/php-master/includes/db/Database.php(1205): DatabaseBase->reportQueryError()
#1 /srv/mediawiki/php-master/includes/db/Database.php(2090): DatabaseBase->query()
#2 /srv/mediawiki/php-master/extensions/Echo/includes/mapper/NotificationMapper.php(40): DatabaseBase->insert()
#3 /srv/mediawiki/php-master/includes/db/Database.php(3460): {closure}()
#4 /srv/mediawiki/php-master/includes/db/Database.php(3421): DatabaseBase->runOnTransactionIdleCallbacks()
#5 /srv/mediawiki/php-master/extensions/Echo/includes/mapper/NotificationMapper.php(48): DatabaseBase->onTransactionIdle()
#6 /srv/mediawiki/php-master/extensions/Echo/model/Notification.php(165): EchoNotificationMapper->insert()
#7 /srv/mediawiki/php-master/extensions/Echo/model/Notification.php(91): EchoNotification->insert()
#8 /srv/mediawiki/php-master/extensions/Echo/Notifier.php(21): EchoNotification::create()
#9 /srv/mediawiki/php-master/extensions/Echo/controller/NotificationController.php(248): EchoNotifier::notifyWithNotification()
#10 /srv/mediawiki/php-master/extensions/Echo/controller/NotificationController.php(86): EchoNotificationController::doNotification()
#11 /srv/mediawiki/php-master/extensions/Echo/jobs/NotificationJob.php(19): EchoNotificationController::notify()
#12 /srv/mediawiki/php-master/includes/jobqueue/JobRunner.php(177): EchoNotificationJob->run()
#13 /srv/mediawiki/rpc/RunJobs.php(42): JobRunner->run()
#14 {main}

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLSobanskiT106854 Duplicate key errors (tracking)
 DeclinedPRODUCTION ERRORNoneT93609 Duplicate key errors in EchoNotificationMapper::insert

Event Timeline

MaxSem created this task.Mar 23 2015, 5:47 PM
MaxSem raised the priority of this task from to Needs Triage.
MaxSem updated the task description. (Show Details)
MaxSem added a project: Notifications.
MaxSem subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 23 2015, 5:47 PM
EBernhardson subscribed.Mar 23 2015, 6:11 PM

Is there any way to get the parameters the job was run with?

EBernhardson triaged this task as Medium priority.Mar 24 2015, 5:45 PM

This might be the same job being executed twice? How can we figure that out?

Legoktm moved this task from Backlog to Needs plan on the Notifications board.Jul 6 2015, 7:35 AM
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptJul 6 2015, 7:35 AM
jcrespo added a project: Wikimedia-production-error.Jul 24 2015, 5:27 PM
jcrespo subscribed.
ori added a parent task: T106854: Duplicate key errors (tracking).Jul 24 2015, 5:29 PM
aaron set Security to None.Jul 24 2015, 9:19 PM
aaron added a subscriber: Legoktm.
aaron subscribed.Jul 24 2015, 10:59 PM

The jobs seem to handle multiple users and inserts. The above trace shows that it's not all in transactional either. If the job fails midway, a retry will start from the beginning, but since the INSERTs cannot handle duplicates it will fail. Usually jobs should be as idempotent as possible.

jcrespo added a comment.EditedJul 26 2015, 3:53 PM

There are 2 UPSERT statements for mysql to make them idempotent:

  • INSERT ... ON DUPLICATE KEY UPDATE
  • REPLACE (INSERT if there is no duplicate, DELETE + INSERT if there is not)

However, I do not know the details of the intended logic- this could be worse than throwing an error (maybe duplicates should be just warnings but make the job be successful).

• demon moved this task from Untriaged to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Jul 29 2015, 4:45 AM
Legoktm renamed this task from Duplicate key errors to Duplicate key errors in EchoNotificationMapper::insert.Aug 9 2015, 10:17 PM
EBernhardson unsubscribed.Jul 18 2018, 6:53 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptJul 18 2018, 6:53 PM
SBisson moved this task from Inbox to Triaged but Future on the Growth-Team board.Jul 20 2018, 5:49 PM
Krinkle removed a project: Collaboration-Team-Triage.Sep 12 2018, 2:19 AM
Krinkle updated the task description. (Show Details)
Krinkle closed this task as Declined.Sep 12 2018, 2:21 AM
Krinkle subscribed.

No matches for 30 days in WMF Logstash for "Duplicate entry" AND "Echo".

jcrespo added a comment.Sep 12 2018, 6:49 AM

I am pretty sure now these inserts do some locking and we get "Deadlock found when trying to get lock; try restarting transaction", which I don't know if it is better or worse :-( Just leaving this as an informative note.

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL