Page MenuHomePhabricator

Fatal error "Lock wait timeout exceeded" from WikiPage::insertRedirectEntry when moving a page
Closed, DuplicatePublicPRODUCTION ERROR

Description

A user on dewiki reports a DBQueryError after move. He moved 3 times with an error and than it works. Sounds like a timeout.

[WPmtzArAEFEAAB9ovAMAAABX] 2017-04-21 06:59:41: Fataler Ausnahmefehler des Typs „DBQueryError“

T150453 fixed something in that area.

Please have look in the logs. Maybe it possible to avoid timeout in that situation. Thanks

Reference: https://de.wikipedia.org/w/index.php?title=Wikipedia:Fragen_zur_Wikipedia&oldid=164795351#datenbankfehler_nach_dem_Verschieben

Event Timeline

Error details:

[WPmtzArAEFEAAB9ovAMAAABX] /w/index.php?title=Spezial:Verschieben&action=submit   DBQueryError from line 1063 of /srv/mediawiki/php-1.29.0-wmf.20/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: INSERT INTO `redirect` (rd_from,rd_namespace,rd_title,rd_fragment,rd_interwiki) VALUES ('9871994','0','Tuukka_Salonen_(Fußballspieler)','','') ON DUPLICATE KEY UPDATE rd_namespace = '0',rd_title = 'Tuukka_Salonen_(Fußballspieler)',rd_fragment = '',rd_interwiki = ''
Function: WikiPage::insertRedirectEntry
Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.192.16.11)

#0 /srv/mediawiki/php-1.29.0-wmf.20/includes/libs/rdbms/database/Database.php(921): Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-1.29.0-wmf.20/includes/libs/rdbms/database/DatabaseMysqlBase.php(1160): Database->query(string, string)
#2 /srv/mediawiki/php-1.29.0-wmf.20/includes/page/WikiPage.php(917): DatabaseMysqlBase->upsert(string, array, array, array, string)
#3 /srv/mediawiki/php-1.29.0-wmf.20/includes/page/WikiPage.php(1283): WikiPage->insertRedirectEntry(Title)
#4 /srv/mediawiki/php-1.29.0-wmf.20/includes/page/WikiPage.php(1243): WikiPage->updateRedirectOn(DatabaseMysqli, Title, NULL)
#5 /srv/mediawiki/php-1.29.0-wmf.20/includes/MovePage.php(596): WikiPage->updateRevisionOn(DatabaseMysqli, Revision, integer)
#6 /srv/mediawiki/php-1.29.0-wmf.20/includes/MovePage.php(271): MovePage->moveToInternal(User, Title, string, boolean, array)
#7 /srv/mediawiki/php-1.29.0-wmf.20/includes/specials/SpecialMovepage.php(590): MovePage->move(User, string, boolean)
#8 /srv/mediawiki/php-1.29.0-wmf.20/includes/specials/SpecialMovepage.php(128): MovePageForm->doSubmit()
#9 /srv/mediawiki/php-1.29.0-wmf.20/includes/specialpage/SpecialPage.php(522): MovePageForm->execute(NULL)
#10 /srv/mediawiki/php-1.29.0-wmf.20/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(NULL)
#11 /srv/mediawiki/php-1.29.0-wmf.20/includes/MediaWiki.php(286): SpecialPageFactory::executePath(Title, RequestContext)
#12 /srv/mediawiki/php-1.29.0-wmf.20/includes/MediaWiki.php(861): MediaWiki->performRequest()
#13 /srv/mediawiki/php-1.29.0-wmf.20/includes/MediaWiki.php(522): MediaWiki->main()
#14 /srv/mediawiki/php-1.29.0-wmf.20/index.php(43): MediaWiki->run()
#15 /srv/mediawiki/w/index.php(3): include(string)
#16 {main}
Umherirrender renamed this task from DBQueryError after move on dewiki to Timeout in WikiPage::insertRedirectEntry after move.Apr 22 2017, 7:07 AM
Umherirrender added subscribers: aaron, Dereckson.

Could be something punctual or he can reproduce it all the time? That server has had 12 errors related RecentChangeSave in the last 24h: https://logstash.wikimedia.org/goto/96d37fb6280d32333f2486eea7efb4ab

It is punctual, the user gets an error 3 times, on the 4. move it works

It is punctual, the user gets an error 3 times, on the 4. move it works

If this is a one time issue, shall we close this and reopen if we see it happening more often?

Feel free to reopen if it happens more often. At least we have now this for tracking.
Thanks!

Feel free to reopen if it happens more often. At least we have now this for tracking.
Thanks!

This depends on user feedback, would be nice to get automatic stats about. But okay, if you think this needs more critical by more reports. Let's wait

I am not saying this is not an issue, what I meant is that if it only happens once it could be just a once time thing. If it happens more often, there might ben an underlying problem there.

Another report, but works later:

[WQIzoQrAIE0AAJxBOI4AAAAT] 2017-04-27 18:08:55: Fataler Ausnahmefehler des Typs „DBQueryError“

From https://de.wikipedia.org/w/index.php?title=Wikipedia:Fragen_zur_Wikipedia&oldid=164971691#Verschiebung_in_den_ANR

jcrespo edited projects, added Wikimedia-production-error; removed DBA.
jcrespo subscribed.

Let's open it, I just think it was misstagged (by assigning it to us, we would normaly close it because that is not an infrastrcture error - normally those things have to go though mediawiki hackers to be triaged. A query failing (#mediwiki-database) is different from the database is down (DBA).

Why that happens have to be researched first.

Used DBA because database admins often can help to see the both queries blocking himself (for deadlocks) or seeing long running queries to change to batching or to use a index (for timeouts).

Thanks to clarify, will change to MediaWiki-libs-Rdbms

I am not saying you did nothing wrong- and yes, we can and do help, I was just justifying that if we see that assigned to us we most likely say "the server is up and running", and close it- think DBA more like machines and mediawiki more like software. Better not tagging it so specific until the real reason is known (which normaly requieres lots of log monitoring and code debugging). People read and monitor MediaWiki-General (I do at least), most people do not read DBA or MediaWiki-libs-Rdbms (or the wrong people do :-P).

Another report:

[WQhSOQrAEFYAAGi4oDsAAABI] 2017-05-02 09:32:58: Fataler Ausnahmefehler des Typs „Wikimedia\Rdbms\DBQueryError“

https://de.wikipedia.org/w/index.php?title=Wikipedia:Fragen_zur_Wikipedia&oldid=165119911#Fataler_Ausnahmefehler

Two from yesterday

[WVjxggpAIDIAAF7OXrEAAAAK] 2017-07-02 13:13:54: Fataler Ausnahmefehler des Typs „Wikimedia\Rdbms\DBQueryError“
[WVjx5ApAEKcAAFy1dIYAAAAT] 2017-07-02 13:15:32: Fataler Ausnahmefehler des Typs „Wikimedia\Rdbms\DBQueryError“

Reported on IRC today:

[01:12]	<Dragonfly6-7> A database query error has occurred. This may indicate a bug in the software.[WV1yJgpAAEEAAJZ@H44AAADB] 2017-07-05 23:11:52: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"

Another from yesterday:

[WV5L9wpAEK4AAEG3X9YAAACC] 2017-07-06 14:41:12: Fataler Ausnahmefehler des Typs „Wikimedia\Rdbms\DBQueryError“

Another one will creating a redirect manually:

[Wa48VwpAEKsAAD2uYAIAAABU] 2017-09-05 05:55:51: Fataler Ausnahmefehler des Typs „Wikimedia\Rdbms\DBQueryError

Umherirrender renamed this task from Timeout in WikiPage::insertRedirectEntry after move to Timeout in WikiPage::insertRedirectEntry after move or redirect creation.Sep 10 2017, 11:21 AM

Still there:

[WoASLwpAMFAAAD1zFxcAAAAG] 2018-02-11 09:52:00: Fataler Ausnahmefehler des Typs „Wikimedia\Rdbms\DBQueryError“

Still seen on wmf.19 as well:

A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: INSERT INTO `redirect` (rd_from,rd_namespace,rd_title,rd_fragment,rd_interwiki) VALUES ('10457846','0','Bumbum_Biggalo','','') ON DUPLICATE KEY UPDATE rd_namespace = '0',rd_title = 'Bumbum_Biggalo',rd_fragment = '',rd_interwiki = ''
Function: WikiPage::insertRedirectEntry
Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.64.48.25)

#0 /srv/mediawiki/php-1.32.0-wmf.19/includes/libs/rdbms/database/Database.php(1428): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.32.0-wmf.19/includes/libs/rdbms/database/Database.php(1198): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.32.0-wmf.19/includes/libs/rdbms/database/DatabaseMysqlBase.php(1365): Wikimedia\Rdbms\Database->query(string, string)
#3 /srv/mediawiki/php-1.32.0-wmf.19/includes/page/WikiPage.php(1035): Wikimedia\Rdbms\DatabaseMysqlBase->upsert(string, array, array, array, string)
#4 /srv/mediawiki/php-1.32.0-wmf.19/includes/page/WikiPage.php(1406): WikiPage->insertRedirectEntry(Title)
#5 /srv/mediawiki/php-1.32.0-wmf.19/includes/page/WikiPage.php(1365): WikiPage->updateRedirectOn(Wikimedia\Rdbms\DatabaseMysqli, Title, NULL)
#6 /srv/mediawiki/php-1.32.0-wmf.19/includes/MovePage.php(589): WikiPage->updateRevisionOn(Wikimedia\Rdbms\DatabaseMysqli, Revision, integer)
#7 /srv/mediawiki/php-1.32.0-wmf.19/includes/MovePage.php(271): MovePage->moveToInternal(User, Title, string, boolean, array)
#8 /srv/mediawiki/php-1.32.0-wmf.19/includes/specials/SpecialMovepage.php(595): MovePage->move(User, string, boolean)
#9 /srv/mediawiki/php-1.32.0-wmf.19/includes/specials/SpecialMovepage.php(128): MovePageForm->doSubmit()
#10 /srv/mediawiki/php-1.32.0-wmf.19/includes/specialpage/SpecialPage.php(569): MovePageForm->execute(NULL)
#11 /srv/mediawiki/php-1.32.0-wmf.19/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#12 /srv/mediawiki/php-1.32.0-wmf.19/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#13 /srv/mediawiki/php-1.32.0-wmf.19/includes/MediaWiki.php(868): MediaWiki->performRequest()
#14 /srv/mediawiki/php-1.32.0-wmf.19/includes/MediaWiki.php(525): MediaWiki->main()
#15 /srv/mediawiki/php-1.32.0-wmf.19/index.php(42): MediaWiki->run()
#16 /srv/mediawiki/w/index.php(3): include(string)
#17 {main}
Krinkle renamed this task from Timeout in WikiPage::insertRedirectEntry after move or redirect creation to "Lock wait timeout exceeded in WikiPage::insertRedirectEntry after move or redirect creation.Sep 5 2018, 2:35 AM
Krinkle renamed this task from "Lock wait timeout exceeded in WikiPage::insertRedirectEntry after move or redirect creation to Fatal error "Lock wait timeout exceeded" from WikiPage::insertRedirectEntry when moving a page.
Krinkle removed a project: MediaWiki-libs-Rdbms.
Krinkle moved this task from Dec2019/1.35.wmf.10+ to Older on the Wikimedia-production-error board.
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM
Krinkle added a subscriber: RoySmith.

"ON DUPLICATE KEY UPDATE" creating a "Lock wait timeout", what a surprise! :-D (you know what I think about ODKU).

For original reporter, explained in plain English: sometimes, certain actions over the same data (e.g. the same page) are incompatible and fail or they would be hang up forever- it doesn't have to be user interactions, it could also be background maintenance tasks. It is impossible to eliminate all possible interactions- but it is our job (developers) to minimize their appearance. You did nothing wrong, and you did ok with retrying (and that is why it worked afterwards). Our job is discovering what interacted here and try to make sure it is very unlikely it happens again. Thanks you very much for the report, flagging them when they happen several times in a row helps us detect them.