Page MenuHomePhabricator
Create Task
Maniphest T186077

'undo' action on SD boards - Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
Closed, ResolvedPublic
Actions

Description

Attempting undo action on betalabs ruwiki brings Database error page.

logstash betalabs displays the following :

January 30th 2018, 15:35:42.000	unique_id:WnEBTgpEE4AAABB@fKwAAAAF reqId:WnEBTgpEE4AAABB@fKwAAAAF server:ru.wikipedia.beta.wmflabs.org db_server:10.68.18.35:3306 wiki:ruwiki channel:DBQuery type:mediawiki error:You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'FORCE INDEX (rc_timestamp) WHERE rc_title = 'Teq8026dm5whantb' AND rc_timestamp' at line 1 (10.68.18.35:3306) http_method:GET @version:1

Details

ProjectBranchLines +/-Subject
mediawiki/extensions/Flowwmf/1.31.0-wmf.20+1 -1AbstractRevision: Specify table in USE INDEX to fix DB error
mediawiki/extensions/Flowmaster+1 -1AbstractRevision: Specify table in USE INDEX to fix DB error
Customize query in gerrit

Related Objects
Search...

Event Timeline

Etonkovidova created this task.Jan 30 2018, 11:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 30 2018, 11:49 PM
Etonkovidova renamed this task from [betalabs] [[WnEBTgpEE4AAABB@fKwAAAAF] 2018-01-30 23:35:42: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" to [betalabs] [WnEBTgpEE4AAABB@fKwAAAAF] 2018-01-30 23:35:42: Fatal exception of type "Wikimedia\Rdbms\DBQueryError".Jan 30 2018, 11:50 PM
• Mattflaschen-WMF triaged this task as High priority.Jan 31 2018, 1:01 AM
SBisson added a subscriber: SBisson.Jan 31 2018, 11:31 AM

I think this is the same thing as T182636: [betalabs] Fatal exception when clicking on deleted topics in View history (Flow\Model\AbstractRevision::getRecentChange)

The FORCE INDEX (rc_timestamp) part is failing in betalabs but working fine locally and in production. Can someone compare the recentchanges indexes between those environments? I don't have access to do it.

Catrope added a subscriber: Catrope.Feb 6 2018, 7:25 PM

The rc_timestamp index exists on both the master and the replica server in beta, in both the enwiki and ruwiki databases.

Etonkovidova added a comment.EditedFeb 6 2018, 9:26 PM

@SBisson, @Catrope - the same error for undo actions on testwiki (wmf.20) SD board.

[Wnob6ApAMFAAAHd@FuUAAACT] /w/index.php?title=User_talk:Etonkovidova&action=undo-edit-header&header_startId=u1k9p0j8ozaraaxr&header_endId=u3ct9o0gog1d4pri   Wikimedia\Rdbms\DBQueryError from line 1187 of /srv/mediawiki/php-1.31.0-wmf.20/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 

Query: SELECT  rc_id,rc_timestamp,rc_user,rc_user_text,rc_namespace,rc_title,rc_minor,rc_bot,rc_new,rc_cur_id,rc_this_oldid,rc_last_oldid,rc_type,rc_source,rc_patrolled,rc_ip,rc_old_len,rc_new_len,rc_deleted,rc_logid,rc_log_type,rc_log_action,rc_params,COALESCE( comment_rc_comment.comment_text, rc_comment ) AS `rc_comment_text`,comment_rc_comment.comment_data AS `rc_comment_data`,comment_rc_comment.comment_id AS `rc_comment_cid`  FROM `recentchanges` LEFT JOIN `comment` `comment_rc_comment` ON ((comment_rc_comment.comment_id = rc_comment_id)) FORCE INDEX (rc_timestamp)  WHERE rc_title = 'Etonkovidova' AND rc_timestamp = '20180206211713' AND rc_namespace = '3'  
Function: Flow\Model\AbstractRevision::getRecentChange
Error: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'FORCE INDEX (rc_timestamp)  WHERE rc_title = 'Etonkovidova' AND rc_timestamp = '' at line 1 (10.64.32.136)
Etonkovidova renamed this task from [betalabs] [WnEBTgpEE4AAABB@fKwAAAAF] 2018-01-30 23:35:42: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" to 'undo' action on SD boards - Fatal exception of type "Wikimedia\Rdbms\DBQueryError".Feb 6 2018, 9:32 PM
Etonkovidova updated the task description. (Show Details)
Jdforrester-WMF added a parent task: T183959: 1.31.0-wmf.20 deployment blockers.Feb 8 2018, 12:54 AM
Jdforrester-WMF added a project: Multi-Content-Revisions.
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

Speculating that this is likely caused by the MCR/RevisionStore rewrite?

Catrope added a comment.Feb 8 2018, 12:59 AM

Full error:

2018-02-08 00:47:02 [WnueBQpAICkAAA-ZvmgAAAAL] mw1320 testwiki 1.31.0-wmf.20 exception ERROR: [WnueBQpAICkAAA-ZvmgAAAAL] /w/index.php?title=Topic:U3m5gyhqh8i8doaw&action=undo-edit-post&topic_startId=u3m5gykc0ml4ov9k&topic_endId=u77cb6noedngq50o   Wikimedia\Rdbms\DBQueryError from line 1187 of /srv/mediawiki/php-1.31.0-wmf.20/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SELECT  rc_id,rc_timestamp,rc_user,rc_user_text,rc_namespace,rc_title,rc_minor,rc_bot,rc_new,rc_cur_id,rc_this_oldid,rc_last_oldid,rc_type,rc_source,rc_patrolled,rc_ip,rc_old_len,rc_new_len,rc_deleted,rc_logid,rc_log_type,rc_log_action,rc_params,COALESCE( comment_rc_comment.comment_text, rc_comment ) AS `rc_comment_text`,comment_rc_comment.comment_data AS `rc_comment_data`,comment_rc_comment.comment_id AS `rc_comment_cid`  FROM `recentchanges` LEFT JOIN `comment` `comment_rc_comment` ON ((comment_rc_comment.comment_id = rc_comment_id)) FORCE INDEX (rc_timestamp)  WHERE rc_title = 'U3m5gyhqh8i8doaw' AND rc_timestamp = '20180208004114' AND rc_namespace = '2600'  
Function: Flow\Model\AbstractRevision::getRecentChange
Error: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'FORCE INDEX (rc_timestamp)  WHERE rc_title = 'U3m5gyhqh8i8doaw' AND rc_timestamp' at line 1 (10.64.16.191)
 {"exception_id":"WnueBQpAICkAAA-ZvmgAAAAL","exception_url":"/w/index.php?title=Topic:U3m5gyhqh8i8doaw&action=undo-edit-post&topic_startId=u3m5gykc0ml4ov9k&topic_endId=u77cb6noedngq50o","caught_by":"mwe_handler"} 
[Exception Wikimedia\Rdbms\DBQueryError] (/srv/mediawiki/php-1.31.0-wmf.20/includes/libs/rdbms/database/Database.php:1187) A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SELECT  rc_id,rc_timestamp,rc_user,rc_user_text,rc_namespace,rc_title,rc_minor,rc_bot,rc_new,rc_cur_id,rc_this_oldid,rc_last_oldid,rc_type,rc_source,rc_patrolled,rc_ip,rc_old_len,rc_new_len,rc_deleted,rc_logid,rc_log_type,rc_log_action,rc_params,COALESCE( comment_rc_comment.comment_text, rc_comment ) AS `rc_comment_text`,comment_rc_comment.comment_data AS `rc_comment_data`,comment_rc_comment.comment_id AS `rc_comment_cid`  FROM `recentchanges` LEFT JOIN `comment` `comment_rc_comment` ON ((comment_rc_comment.comment_id = rc_comment_id)) FORCE INDEX (rc_timestamp)  WHERE rc_title = 'U3m5gyhqh8i8doaw' AND rc_timestamp = '20180208004114' AND rc_namespace = '2600'  
Function: Flow\Model\AbstractRevision::getRecentChange
Error: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'FORCE INDEX (rc_timestamp)  WHERE rc_title = 'U3m5gyhqh8i8doaw' AND rc_timestamp' at line 1 (10.64.16.191)

  #0 /srv/mediawiki/php-1.31.0-wmf.20/includes/libs/rdbms/database/Database.php(998): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
  #1 /srv/mediawiki/php-1.31.0-wmf.20/includes/libs/rdbms/database/Database.php(1382): Wikimedia\Rdbms\Database->query(string, string)
  #2 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Model/AbstractRevision.php(861): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
  #3 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Formatter/RevisionViewFormatter.php(133): Flow\Model\AbstractRevision->getRecentChange()
  #4 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Formatter/RevisionViewFormatter.php(44): Flow\Formatter\RevisionViewFormatter->buildLinks(Flow\Formatter\FormatterRow, Flow\View)
  #5 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Formatter/RevisionUndoViewFormatter.php(49): Flow\Formatter\RevisionViewFormatter->formatApi(Flow\Formatter\FormatterRow, Flow\View)
  #6 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Block/Topic.php(714): Flow\Formatter\RevisionUndoViewFormatter->formatApi(Flow\Formatter\FormatterRow, Flow\Formatter\FormatterRow, Flow\Formatter\FormatterRow, Flow\View)
  #7 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Block/Topic.php(527): Flow\Block\TopicBlock->renderUndoApi(array)
  #8 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/View.php(217): Flow\Block\TopicBlock->renderApi(array)
  #9 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/View.php(69): Flow\View->buildApiResponse(Flow\WorkflowLoader, array, string, array)
  #10 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Actions/Action.php(112): Flow\View->show(Flow\WorkflowLoader, string)
  #11 /srv/mediawiki/php-1.31.0-wmf.20/extensions/Flow/includes/Actions/Action.php(50): Flow\Actions\FlowAction->showForAction(string)
  #12 /srv/mediawiki/php-1.31.0-wmf.20/includes/MediaWiki.php(500): Flow\Actions\FlowAction->show()
  #13 /srv/mediawiki/php-1.31.0-wmf.20/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
  #14 /srv/mediawiki/php-1.31.0-wmf.20/includes/MediaWiki.php(858): MediaWiki->performRequest()
  #15 /srv/mediawiki/php-1.31.0-wmf.20/includes/MediaWiki.php(524): MediaWiki->main()
  #16 /srv/mediawiki/php-1.31.0-wmf.20/index.php(42): MediaWiki->run()
  #17 /srv/mediawiki/w/index.php(3): include(string)
  #18 {main}

Note that this isn't a missing index or anything, there truly is a syntax error in the SQL: SELECT stuff FROM recentchanges LEFT JOIN comment comment_rc_comment ON (stuff) FORCE INDEX (rc_timestamp) is incorrect, it should be SELECT stuff FROM recentchanges FORCE INDEX (rc_timestamp) LEFT JOIN ....

The code that causes this query uses RecentChange::getQueryInfo() to build the query. The output of that on testwiki is:

> var_dump(RecentChange::getQueryInfo())
array(3) {
  ["tables"]=>
  array(2) {
    [0]=>
    string(13) "recentchanges"
    ["comment_rc_comment"]=>
    string(7) "comment"
  }
  ["fields"]=>
  array(26) {
    ....
  }
  ["joins"]=>
  array(1) {
    ["comment_rc_comment"]=>
    array(2) {
      [0]=>
      string(9) "LEFT JOIN"
      [1]=>
      string(45) "comment_rc_comment.comment_id = rc_comment_id"
    }
  }
}

The join wasn't there until recently. The options that we pass into the query are ['USE INDEX' => 'rc_timestamp'] which is wrong now that there are multiple tables. It has to be a per-table nested array now. Patch incoming.

gerritbot added a subscriber: gerritbot.Feb 8 2018, 1:01 AM

Change 408958 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/extensions/Flow@master] AbstractRevision: Specify table in USE INDEX to fix DB error

https://gerrit.wikimedia.org/r/408958

gerritbot added a project: Patch-For-Review.Feb 8 2018, 1:01 AM
Catrope edited projects, added Collaboration-Team-Triage (Collab-Team-This-Quarter); removed Multi-Content-Revisions, Collaboration-Team-Triage.Feb 8 2018, 1:02 AM
Catrope moved this task from Untriaged to Needs Review on the Collaboration-Team-Triage (Collab-Team-This-Quarter) board.
demon added a subscriber: demon.Feb 8 2018, 1:58 AM

T186764: refreshLinks/jobqueue issues in wmf.20 causing MW-reported replag related?

gerritbot added a comment.Feb 8 2018, 3:09 AM

Change 408958 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] AbstractRevision: Specify table in USE INDEX to fix DB error

https://gerrit.wikimedia.org/r/408958

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-02-13 (1.31.0-wmf.21)).Feb 8 2018, 4:00 AM
Jdforrester-WMF added a comment.Feb 8 2018, 4:17 AM

Fixed in master - see https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Topic:U77o0uiommsw9clr&action=history for a successfully undone edit. Can’t tell from here whether it’s the same as that DB fatal spike, but it seems unlikely (undos on StructuredDiscussions are not that frequent).

gerritbot added a comment.Feb 8 2018, 4:18 AM

Change 408966 had a related patch set uploaded (by Jforrester; owner: Catrope):
[mediawiki/extensions/Flow@wmf/1.31.0-wmf.20] AbstractRevision: Specify table in USE INDEX to fix DB error

https://gerrit.wikimedia.org/r/408966

SBisson moved this task from Needs Review to QA Review on the Collaboration-Team-Triage (Collab-Team-This-Quarter) board.Feb 8 2018, 10:43 AM
SBisson mentioned this in T182636: [betalabs] Fatal exception when clicking on deleted topics in View history (Flow\Model\AbstractRevision::getRecentChange).
gerritbot added a comment.Feb 8 2018, 7:12 PM

Change 408966 merged by jenkins-bot:
[mediawiki/extensions/Flow@wmf/1.31.0-wmf.20] AbstractRevision: Specify table in USE INDEX to fix DB error

https://gerrit.wikimedia.org/r/408966

Stashbot added a subscriber: Stashbot.Feb 8 2018, 7:24 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-08T19:24:12Z] <catrope@tin> Synchronized php-1.31.0-wmf.20/extensions/Flow/includes/Model/AbstractRevision.php: T186077 (duration: 01m 11s)

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)); removed MW-1.31-release-notes (WMF-deploy-2018-02-13 (1.31.0-wmf.21)).Feb 8 2018, 8:00 PM
Jdforrester-WMF closed this task as Resolved.Feb 8 2018, 8:14 PM
Jdforrester-WMF assigned this task to Catrope.
Jdforrester-WMF removed a project: Patch-For-Review.

Tested on wmf.20 production; worked great: https://test.wikipedia.org/w/index.php?title=Topic:U795k02bwh6ao6v1&action=history

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL