Page MenuHomePhabricator

CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks
Closed, DeclinedPublicPRODUCTION ERROR

Description

2015-12-10 09:24:46 mw1004 metawiki exception ERROR: [c11dec9e] /rpc/RunJobs.php?wiki=metawiki&type=CentralAuthCreateLocalAccountJob&maxtime=60&maxmem=300M   DBQueryError from line 1090 of /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO `echo_notification` (notification_event,notification_user,notification_timestamp,notification_read_timestamp,notification_bundle_base,notification_bundle_hash,notification_bundle_display_hash) VALUES ('2436051','10299427','20151210092446',NULL,'1','35b61ec771e85e8dc88af5a2506502cf','4c306688bbb1d653f3876a560cbf1ed3')
Function: EchoNotificationMapper::insert
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.22)
 {"exception_id":"c11dec9e"} 
[Exception DBQueryError] (/srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php:1090) A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO `echo_notification` (notification_event,notification_user,notification_timestamp,notification_read_timestamp,notification_bundle_base,notification_bundle_hash,notification_bundle_display_hash) VALUES ('2436051','10299427','20151210092446',NULL,'1','35b61ec771e85e8dc88af5a2506502cf','4c306688bbb1d653f3876a560cbf1ed3')
Function: EchoNotificationMapper::insert
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.22)

  #0 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(1047): DatabaseBase->reportQueryError(string, integer, string, string, boolean)
  #1 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(1935): DatabaseBase->query(string, string)
  #2 /srv/mediawiki/php-1.27.0-wmf.8/extensions/Echo/includes/mapper/NotificationMapper.php(56): DatabaseBase->insert(string, array, string)
  #3 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(3310): Closure$EchoNotificationMapper::insert()
  #4 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(3566): DatabaseBase->runOnTransactionIdleCallbacks()
  #5 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LoadBalancer.php(1050): DatabaseBase->commit(string, string)
  #6 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactory.php(194): LoadBalancer->commitMasterChanges()
  #7 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactoryMulti.php(394): Closure$LBFactory::forEachLBCallMethod(LoadBalancer, string, array)
  #8 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactory.php(195): LBFactoryMulti->forEachLB(Closure$LBFactory::forEachLBCallMethod;1521762499, array)
  #9 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactory.php(212): LBFactory->forEachLBCallMethod(string)
  #10 /srv/mediawiki/php-1.27.0-wmf.8/includes/deferred/DeferredUpdates.php(152): LBFactory->commitMasterChanges()
  #11 /srv/mediawiki/php-1.27.0-wmf.8/includes/deferred/DeferredUpdates.php(88): DeferredUpdates::execute(array, string)
  #12 /srv/mediawiki/php-1.27.0-wmf.8/includes/jobqueue/JobRunner.php(184): DeferredUpdates::doUpdates()
  #13 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run(array)
  #14 {main}

Event Timeline

Legoktm created this task.Dec 10 2015, 10:46 PM
Legoktm raised the priority of this task from to Needs Triage.
Legoktm updated the task description. (Show Details)
Legoktm added subscribers: Legoktm, Tgr.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

@jcrespo Any idea what may be causing this deadlock?

jcrespo added a comment.EditedJul 14 2016, 10:46 AM

deadlocks are caused by inter-locks between InnoDB transactions, which want to edit each other's locked rows. Check what rows had been locked on that transaction before the error and ether avoid excessive locking, shorter (in time) or smaller (in size) transactions (commit more frequently), or too much concurrency happening at the same time (are you trying to insert too many things at the same time?).

Deadlocks by themselves are not problematic (unless the rate is really high)- the application should capture the errors, maybe wait a random time and retry them (unless the problem is that the transaction takes too long).

This is either likely related or caused by the same issue: T139970

ori moved this task from Inbox to Radar on the Performance-Team board.Jul 18 2016, 6:43 PM
Krinkle set Security to None.

Is this still relevant?

Tgr added a comment.Oct 9 2017, 7:29 PM

It's still happening a few dozen times a week.

1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:45 AM
Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 10:57 AM
Krinkle closed this task as Declined.Sep 12 2018, 11:08 PM
Krinkle added a subscriber: Krinkle.

Not seen in Logstash for at least 30 days.

Restricted Application added a project: Growth-Team. · View Herald TranscriptSep 12 2018, 11:08 PM
Krinkle changed Risk Rating from N/A to default.
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM