Page MenuHomePhabricator

CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks
Closed, DeclinedPublicPRODUCTION ERROR

Description

2015-12-10 09:24:46 mw1004 metawiki exception ERROR: [c11dec9e] /rpc/RunJobs.php?wiki=metawiki&type=CentralAuthCreateLocalAccountJob&maxtime=60&maxmem=300M   DBQueryError from line 1090 of /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO `echo_notification` (notification_event,notification_user,notification_timestamp,notification_read_timestamp,notification_bundle_base,notification_bundle_hash,notification_bundle_display_hash) VALUES ('2436051','10299427','20151210092446',NULL,'1','35b61ec771e85e8dc88af5a2506502cf','4c306688bbb1d653f3876a560cbf1ed3')
Function: EchoNotificationMapper::insert
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.22)
 {"exception_id":"c11dec9e"} 
[Exception DBQueryError] (/srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php:1090) A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO `echo_notification` (notification_event,notification_user,notification_timestamp,notification_read_timestamp,notification_bundle_base,notification_bundle_hash,notification_bundle_display_hash) VALUES ('2436051','10299427','20151210092446',NULL,'1','35b61ec771e85e8dc88af5a2506502cf','4c306688bbb1d653f3876a560cbf1ed3')
Function: EchoNotificationMapper::insert
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.22)

  #0 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(1047): DatabaseBase->reportQueryError(string, integer, string, string, boolean)
  #1 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(1935): DatabaseBase->query(string, string)
  #2 /srv/mediawiki/php-1.27.0-wmf.8/extensions/Echo/includes/mapper/NotificationMapper.php(56): DatabaseBase->insert(string, array, string)
  #3 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(3310): Closure$EchoNotificationMapper::insert()
  #4 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/Database.php(3566): DatabaseBase->runOnTransactionIdleCallbacks()
  #5 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LoadBalancer.php(1050): DatabaseBase->commit(string, string)
  #6 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactory.php(194): LoadBalancer->commitMasterChanges()
  #7 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactoryMulti.php(394): Closure$LBFactory::forEachLBCallMethod(LoadBalancer, string, array)
  #8 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactory.php(195): LBFactoryMulti->forEachLB(Closure$LBFactory::forEachLBCallMethod;1521762499, array)
  #9 /srv/mediawiki/php-1.27.0-wmf.8/includes/db/loadbalancer/LBFactory.php(212): LBFactory->forEachLBCallMethod(string)
  #10 /srv/mediawiki/php-1.27.0-wmf.8/includes/deferred/DeferredUpdates.php(152): LBFactory->commitMasterChanges()
  #11 /srv/mediawiki/php-1.27.0-wmf.8/includes/deferred/DeferredUpdates.php(88): DeferredUpdates::execute(array, string)
  #12 /srv/mediawiki/php-1.27.0-wmf.8/includes/jobqueue/JobRunner.php(184): DeferredUpdates::doUpdates()
  #13 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run(array)
  #14 {main}

Event Timeline

Legoktm raised the priority of this task from to Needs Triage.
Legoktm updated the task description. (Show Details)
Legoktm added subscribers: Legoktm, Tgr.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

deadlocks are caused by inter-locks between InnoDB transactions, which want to edit each other's locked rows. Check what rows had been locked on that transaction before the error and ether avoid excessive locking, shorter (in time) or smaller (in size) transactions (commit more frequently), or too much concurrency happening at the same time (are you trying to insert too many things at the same time?).

Deadlocks by themselves are not problematic (unless the rate is really high)- the application should capture the errors, maybe wait a random time and retry them (unless the problem is that the transaction takes too long).

This is either likely related or caused by the same issue: T139970

Krinkle added a subscriber: Krinkle.

Not seen in Logstash for at least 30 days.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM