Page MenuHomePhabricator

internal_api_error_DBQueryError: Database query error while (mass) deleting file over api
Closed, ResolvedPublic

Description

I get such error since a few days when doing mass deletions:

API request failed (internal_api_error_DBQueryError): [0b608d20] Database query error

Event Timeline

Steinsplitter raised the priority of this task from to Normal.
Steinsplitter updated the task description. (Show Details)
Steinsplitter added a subscriber: Steinsplitter.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 10 2015, 4:54 PM
Steinsplitter set Security to None.

And sometimes file links remaining blue, after purging by hand they become red. Not sure if this is related.

Steinsplitter moved this task from Incoming to Backlog on the Commons board.May 10 2015, 4:57 PM
hoo added a subscriber: hoo.May 10 2015, 5:00 PM

Backtrace (page/ revision id removed):

2015-05-10 16:52:01 mw1208 commonswiki exception INFO: [0b608d20] /w/api.php   DBQueryError from line 1276 of /srv/mediawiki/php-1.26wmf4/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: SELECT  rev_id,rev_page,rev_text_id,rev_timestamp,rev_comment,rev_user_text,rev_user,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,rev_content_format,rev_content_model,page_namespace,page_title,page_id,page_latest,page_is_redirect,page_len,user_name  FROM `revision` INNER JOIN `page` ON ((page_id = rev_page)) LEFT JOIN `user` ON ((rev_user != 0) AND (user_id = rev_user))  WHERE page_id = '...' AND rev_id = '....'  LIMIT 1   FOR UPDATE
Function: Revision::fetchFromConds
Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.64.16.29)

#0 /srv/mediawiki/php-1.26wmf4/includes/db/Database.php(1231): DatabaseBase->reportQueryError()
#1 /srv/mediawiki/php-1.26wmf4/includes/db/Database.php(1756): DatabaseBase->query()
#2 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(400): DatabaseBase->select()
#3 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(342): Revision::fetchFromConds()
#4 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(312): Revision::loadFromConds()
#5 /srv/mediawiki/php-1.26wmf4/includes/Revision.php(151): Revision::newFromConds()
#6 /srv/mediawiki/php-1.26wmf4/includes/page/WikiPage.php(620): Revision::newFromPageId()
#7 /srv/mediawiki/php-1.26wmf4/includes/page/WikiPage.php(661): WikiPage->loadLastEdit()
#8 /srv/mediawiki/php-1.26wmf4/includes/page/WikiPage.php(2796): WikiPage->getContent()
#9 /srv/mediawiki/php-1.26wmf4/includes/FileDeleteForm.php(193): WikiPage->doDeleteArticleReal()
#10 /srv/mediawiki/php-1.26wmf4/includes/api/ApiDelete.php(174): FileDeleteForm::doDelete()
#11 /srv/mediawiki/php-1.26wmf4/includes/api/ApiDelete.php(61): ApiDelete::deleteFile()
#12 /srv/mediawiki/php-1.26wmf4/includes/api/ApiMain.php(1131): ApiDelete->execute()
#13 /srv/mediawiki/php-1.26wmf4/includes/api/ApiMain.php(469): ApiMain->executeAction()
#14 /srv/mediawiki/php-1.26wmf4/includes/api/ApiMain.php(442): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.26wmf4/api.php(90): ApiMain->execute()
#16 /srv/mediawiki/w/api.php(3): include()
#17 {main} {"private":false}
Steinsplitter raised the priority of this task from Normal to High.May 11 2015, 10:31 AM

API request failed (internal_api_error_DBQueryError): [e6308580] Database query error
API request failed (internal_api_error_DBQueryError): [c88d0938] Database query error
API request failed (internal_api_error_DBQueryError): [a3914641] Database query error
API request failed (internal_api_error_DBQueryError): [c6308edf] Database query error
API request failed (internal_api_error_DBQueryError): [f765ae82] Database query error
(.....)

Umherirrender renamed this task from internal_api_error_DBQueryError: Database query error to internal_api_error_DBQueryError: Database query error while (mass) deleting file over api.May 29 2015, 6:11 PM
Umherirrender added subscribers: aaron, Springle.

Change 214684 had a related patch set uploaded (by Aaron Schulz):
Reduce page deletion lock contention

https://gerrit.wikimedia.org/r/214684

Change 214684 merged by jenkins-bot:
Reduce page deletion lock contention

https://gerrit.wikimedia.org/r/214684

chasemp assigned this task to aaron.Jun 2 2015, 5:22 PM
chasemp added a subscriber: chasemp.

@aaron, did your patch address this successfully?

I'm not sure where this issue stands, can you throw some commentary in here?

aaron added a comment.Jun 2 2015, 5:30 PM

The query mentioned by hoo will no longer be "FOR UPDATE", so that error can't happen after the change is deployed. It would still be good to keep an eye on any page table related locks.

aaron closed this task as Resolved.Jun 2 2015, 5:31 PM

Change 215379 had a related patch set uploaded (by Aaron Schulz):
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215379

Change 215379 merged by jenkins-bot:
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215379

Change 215388 had a related patch set uploaded (by Aaron Schulz):
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215388

Change 215388 merged by jenkins-bot:
Fixed $flags bit operation precedence fail in User::loadFromDatabase()

https://gerrit.wikimedia.org/r/215388

2015-06-13 01:01:16 mw1223 commonswiki exception INFO: [5fb8a4d0] /w/api.php   DBQueryError from line 1276 of /srv/mediawiki/php-1.26wmf9/includes/db/Database.php: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT  INTO `recentchanges` (rc_timestamp,rc_namespace,rc_title,rc_type,rc_source,rc_minor,rc_cur_id,rc_user,rc_user_text,rc_comment,rc_this_oldid,rc_last_oldid,rc_bot,rc_ip,rc_patrolled,rc_new,rc_old_len,rc_new_len,rc_deleted,rc_logid,rc_log_type,rc_log_action,rc_params,rc_id) VALUES (redacted)
Function: RecentChange::save
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.29)

#0 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(1231): DatabaseBase->reportQueryError()
#1 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(2117): DatabaseBase->query()
#2 /srv/mediawiki/php-1.26wmf9/includes/changes/RecentChange.php(306): DatabaseBase->insert()
#3 /srv/mediawiki/php-1.26wmf9/includes/logging/LogEntry.php(629): RecentChange->save()
#4 /srv/mediawiki/php-1.26wmf9/includes/page/WikiPage.php(2867): ManualLogEntry->publish()
#5 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(3515): Closure$WikiPage::doDeleteArticleReal()
#6 /srv/mediawiki/php-1.26wmf9/includes/db/Database.php(3724): DatabaseBase->runOnTransactionPreCommitCallbacks()
#7 /srv/mediawiki/php-1.26wmf9/includes/FileDeleteForm.php(199): DatabaseBase->commit()
#8 /srv/mediawiki/php-1.26wmf9/includes/api/ApiDelete.php(174): FileDeleteForm::doDelete()
#9 /srv/mediawiki/php-1.26wmf9/includes/api/ApiDelete.php(61): ApiDelete::deleteFile()
#10 /srv/mediawiki/php-1.26wmf9/includes/api/ApiMain.php(1130): ApiDelete->execute()
#11 /srv/mediawiki/php-1.26wmf9/includes/api/ApiMain.php(469): ApiMain->executeAction()
#12 /srv/mediawiki/php-1.26wmf9/includes/api/ApiMain.php(442): ApiMain->executeActionWithErrorHandling()
#13 /srv/mediawiki/php-1.26wmf9/api.php(88): ApiMain->execute()
#14 /srv/mediawiki/w/api.php(3): include()
#15 {main} {"private":false}

Translations always trigger two edits at the same time. Just now one of them failed with

Saving the translation failed: [42491262] Database query error

leaving the translation in some "undefined" state.

Change 218557 had a related patch set uploaded (by Aaron Schulz):
Reduce RC table contention for file deletion

https://gerrit.wikimedia.org/r/218557

Change 218557 merged by jenkins-bot:
Reduce RC table contention for file deletion

https://gerrit.wikimedia.org/r/218557

Ankry added a subscriber: Ankry.Jun 18 2015, 5:33 AM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 26 2015, 10:43 AM
TTO added a subscriber: TTO.Jun 27 2015, 12:18 PM

Fixed? Can close?

aaron closed this task as Resolved.Jul 2 2015, 11:37 PM

Error while moving the page.
A detailed description of the error is shown below:
API request failed (internal_api_error_DBQueryError): [e96e342a] Database query error <i>at Sun, 05 Jul 2015 09:34:10 GMT</i> <u>served by mw1189</u>

Is this related?

hoo added a comment.Jul 5 2015, 10:44 AM

Error while moving the page.
A detailed description of the error is shown below:
API request failed (internal_api_error_DBQueryError): [e96e342a] Database query error <i>at Sun, 05 Jul 2015 09:34:10 GMT</i> <u>served by mw1189</u>
Is this related?

No, it's not, that was about the image table… you can open a separate bug for that, if you want.