Page MenuHomePhabricator
Create Task
Maniphest T121644

Fail to load more topics above current topic due to pseudo-column being queried
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Steps:

  1. Go to a large board ( http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA )
  2. In the ToC, scroll for a while and click on a topic in order to navigate to a topic that is not yet loaded
  3. When the topic is loaded and scrolled to, scroll up just a little bit to trigger backfilling of topics above the one you clicked
  4. Database query error

This is the error when reproduced locally:

[d570267b] Exception Caught: A database error has occurred. Did you forget to run maintenance/update.php after upgrading? 
See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script 
Query: SELECT * FROM `flow_topic_list` WHERE topic_list_id = '�=~��?@��' ORDER BY workflow_last_update_timestamp desc LIMIT 11 
Function: Flow\Data\Storage\BasicDbStorage::find (flow_topic_list) 
Error: 1054 Unknown column 'workflow_last_update_timestamp' in 'order clause' (127.0.0.1)

Details

SubjectRepoBranchLines +/-
Don't attempt to do reverse pagination in flow-loadmoremediawiki/extensions/Flowmaster+8 -0
Make sure topiclist queries always join against workflow tablemediawiki/extensions/Flowwmf/1.29.0-wmf.17+23 -68
Make sure topiclist queries always join against workflow tablemediawiki/extensions/Flowmaster+23 -68
Customize query in gerrit

Related Objects
Search...

Event Timeline

SBisson created this task.Dec 16 2015, 3:47 PM
SBisson raised the priority of this task from to Needs Triage.
SBisson updated the task description. (Show Details)
SBisson added projects: StructuredDiscussions, Collaboration-Team-Archive-2015-2016.
SBisson added subscribers: SBisson, • Mattflaschen-WMF, Catrope, Etonkovidova.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 16 2015, 3:47 PM
• Mattflaschen-WMF triaged this task as High priority.Dec 16 2015, 9:08 PM
jcrespo added a project: Wikimedia-production-error.EditedDec 17 2015, 12:02 PM
jcrespo subscribed.

I can see this error on mediawikiwiki logs, generated by api calls (I arrived here independently):

{
  "_index": "logstash-2015.12.17",
  "_type": "mediawiki",
  "_id": "AVGvy9OKMRv_gmyxU30n",
  "_score": null,
  "_source": {
    "message": "Flow\\Data\\Storage\\BasicDbStorage::find (flow_topic_list)\t10.64.16.20\t1054\tUnknown column 'workflow_last_update_timestamp' in 'order clause' (10.64.16.20)\tSELECT  *  FROM `flow_topic_list`   WHERE topic_list_id = '\u00059ÕŸ*žŠ1ùå\u0005'  ORDER BY workflow_last_update_timestamp desc LIMIT 11  ",
    "@version": 1,
    "@timestamp": "2015-12-17T11:55:48.000Z",
    "type": "mediawiki",
    "host": "mw1201",
    "level": "ERROR",
    "tags": [
      "syslog",
      "es",
      "es",
      "normalized_message_trimmed"
    ],
    "channel": "wfLogDBError",
    "url": "/w/api.php?action=flow&format=json&page=Extension_talk%3ACollection&submodule=view-topiclist&vtloffset-dir=rev&vtllimit=10&vtloffset=20150504134619&vtlsortby=updated",
    "ip": "10.64.32.106",
    "http_method": "GET",
    "server": "www.mediawiki.org",
    "referrer": "https://www.mediawiki.org/wiki/Extension_talk:Collection",
    "uid": "fee722d",
    "process_id": 31016,
    "wiki": "mediawikiwiki",
    "db_server": "10.64.16.20",
    "db_name": "flowdb",
    "db_user": "wikiuser",
    "method": "DatabaseBase::reportQueryError",
    "errno": 1054,
    "error": "Unknown column 'workflow_last_update_timestamp' in 'order clause' (10.64.16.20)",
    "sql1line": "SELECT  *  FROM `flow_topic_list`   WHERE topic_list_id = '\u00059ÕŸ*žŠ1ùå\u0005'  ORDER BY workflow_last_update_timestamp desc LIMIT 11  ",
    "fname": "Flow\\Data\\Storage\\BasicDbStorage::find (flow_topic_list)",
    "normalized_message": "Flow\\Data\\Storage\\BasicDbStorage::find (flow_topic_list)\t10.64.16.20\t1054\tUnknown column 'workflow_last_update_timestamp' in 'order clause' (10.64.16.20)\tSELECT  *  FROM `flow_topic_list`   WHERE topic_list_id = '\u00059ÕŸ*žŠ1ùå\u0005'  ORDER BY workflow_last_updat"
  },
  "sort": [
    1450353348000
  ]
}

Only 7 failures in the last day, though.

• demon moved this task from Untriaged to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Dec 17 2015, 4:11 PM
Catrope moved this task from Untriaged to Ready for pickup on the Collaboration-Team-Archive-2015-2016 board.Dec 19 2015, 1:22 AM
hashar added subscribers: • Elitre, hashar.Feb 2 2016, 9:00 AM

Been reported yesterday by @Elitre while browsing the office wiki:

[8f733e14] Database query error while scrolling https://office.wikimedia.org/wiki/Village_pump . I should join Collaboration's #, I guess they have one.)

Still an order by close referencing the non existent workflow_last_update_timestamp field

Function: Flow\Data\Storage\BasicDbStorage::find (flow_topic_list)
Error: 1054 Unknown column 'workflow_last_update_timestamp' in 'order clause' (10.64.16.24)
• Mattflaschen-WMF renamed this task from Fail to load more topics above current topic to Fail to load more topics above current topic due to psuedo-column being queried.Feb 4 2016, 9:00 PM
• Mattflaschen-WMF set Security to None.
• Mattflaschen-WMF merged a task: T125872: Flow\Data\Storage\BasicDbStorage::find Unknown column 'workflow_last_update_timestamp' in 'order clause'.
• jmatazzoni moved this task from Ready for pickup to RFP (Flow) on the Collaboration-Team-Archive-2015-2016 board.Apr 6 2016, 10:33 PM
• jmatazzoni edited projects, added Collab-Team-2016-Apr-Jun-Q4; removed Collaboration-Team-Archive-2015-2016.Apr 11 2016, 11:33 PM
• jmatazzoni moved this task from Untriaged to Ready for Pickup on the Collab-Team-2016-Apr-Jun-Q4 board.Apr 11 2016, 11:54 PM
jcrespo added a comment.May 19 2016, 8:52 AM
{
  "_index": "logstash-2016.05.19",
  "_type": "mediawiki",
  "_id": "AVTIL9cMDxp7yus2XfK0",
  "_score": null,
  "_source": {
    "message": "Flow\\Data\\Storage\\BasicDbStorage::find (flow_topic_list)\t10.64.16.18\t1054\tUnknown column 'workflow_last_update_timestamp' in 'order clause' (10.64.16.18)\tSELECT  *  FROM `flow_topic_list`   WHERE topic_list_id = '\u0005Jšä颪\u0018Ӓ†'  ORDER BY workflow_last_update_timestamp desc LIMIT 11  ",
    "@version": 1,
    "@timestamp": "2016-05-19T08:44:18.000Z",
    "type": "mediawiki",
    "host": "mw1139",
    "level": "ERROR",
    "tags": [
      "syslog",
      "es",
      "es"
    ],
    "channel": "wfLogDBError",
    "normalized_message": "{fname}\t{db_server}\t{errno}\t{error}\t{sql1line}",
    "url": "/w/api.php?action=flow&format=json&page=Discussion_utilisateur%3AH4stings&submodule=view-topiclist&vtloffset-dir=rev&vtllimit=10&vtloffset=20160309223857&vtlsortby=updated",
    "ip": "10.64.0.103",
    "http_method": "GET",
    "server": "fr.wikipedia.org",
    "referrer": "https://fr.wikipedia.org/wiki/Discussion_utilisateur:H4stings",
    "unique_id": "Vz183wpAEHcAAEsqjVcAAAAX",
    "wiki": "frwiki",
    "mwversion": "1.28.0-wmf.1",
    "reqId": "Vz183wpAEHcAAEsqjVcAAAAX",
    "db_server": "10.64.16.18",
    "db_name": "flowdb",
    "db_user": "wikiuser",
    "method": "DatabaseBase::reportQueryError",
    "errno": 1054,
    "error": "Unknown column 'workflow_last_update_timestamp' in 'order clause' (10.64.16.18)",
    "sql1line": "SELECT  *  FROM `flow_topic_list`   WHERE topic_list_id = '\u0005Jšä颪\u0018Ӓ†'  ORDER BY workflow_last_update_timestamp desc LIMIT 11  ",
    "fname": "Flow\\Data\\Storage\\BasicDbStorage::find (flow_topic_list)"
  },
  "sort": [
    1463647458000
  ]
}
• Elitre renamed this task from Fail to load more topics above current topic due to psuedo-column being queried to Fail to load more topics above current topic due to pseudo-column being queried.May 19 2016, 11:11 AM
jcrespo added a comment.EditedMay 19 2016, 11:36 AM

Sorry, I think my comment got lost, and I only posted the error. I meant that I saw a small, but noticeable spike of this errors on frwiki, and I showed an example including a timestamp if you want to look at the rest.

hashar unsubscribed.May 19 2016, 1:17 PM
• Mattflaschen-WMF added a comment.May 19 2016, 9:52 PM

Probably will be fixed as part of T120009: Flow: Use WAN cache delete() and replica-based filling to avoid merge()

• jmatazzoni moved this task from Ready for Pickup to Untriaged on the Collab-Team-2016-Apr-Jun-Q4 board.Jul 11 2016, 9:22 PM
• jmatazzoni removed a project: Collab-Team-2016-Apr-Jun-Q4.Jul 12 2016, 4:36 PM
• jmatazzoni added a project: Collab-Team-Q1-July-Sep-2016.
• jmatazzoni moved this task from Untriaged to Ready for Pickup on the Collab-Team-Q1-July-Sep-2016 board.
Etonkovidova added a comment.Aug 16 2016, 9:03 PM

Checked in 1.28.0-alpha (b5e308c). The page http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA consistently shows 'Database query error' , for exmple:

{servedby: "deployment-mediawiki02",…}
error
:
{code: "internal_api_error_DBQueryError", info: "[V7N@SwpEEH8AAE1DMKMAAAAG] Database query error",…}
*
:
"#0 /srv/mediawiki/php-master/includes/db/Database.php(911): 
DatabaseBase->reportQueryError(string, integer, string, string, 
boolean)
#1 /srv/mediawiki/php-
master/includes/db/Database.php(1278): DatabaseBase->query(string, string)
#2 /srv/mediawiki/php-master/extensions/Flow/includes/Data/Storage/BasicDbStorage.php(148): DatabaseBase->select(string, string, array, string, array)
#3 /srv/mediawiki/php-master/extensions/Flow/includes/Data/Storage/BasicDbStorage.php(163): Flow\Data\Storage\BasicDbStorage->find(array, array)
#4 /srv/mediawiki/php-master/extensions/Flow/includes/Data/Storage/BasicDbStorage.php(180): Flow\Data\Storage\BasicDbStorage->fallbackFindMulti(array, array)
#5 /srv/mediawiki/php-master/extensions/Flow/includes/Data/ObjectLocator.php(103): Flow\Data\Storage\BasicDbStorage->findMulti(array, array)
#6 /srv/mediawiki/php-master/extensions/Flow/includes/Data/ObjectLocator.php(64): Flow\Data\ObjectLocator->findMulti(array, array)
#7 /srv/mediawiki/php-master/extensions/Flow/includes/Data/Pager/Pager.php(130): Flow\Data\ObjectLocator->find(array, array)
#8 /srv/mediawiki/php-master/extensions/Flow/includes/Block/TopicList.php(479): Flow\Data\Pager\Pager->getPage(Closure$Flow\Block\TopicListBlock::getPage;30357559)
#9 /srv/mediawiki/php-master/extensions/Flow/includes/Block/TopicList.php(266): Flow\Block\TopicListBlock->getPage(array)
#10 /srv/mediawiki/php-master/extensions/Flow/includes/Api/ApiFlowBaseGet.php(33): Flow\Block\TopicListBlock->renderApi(array)
#11 /srv/mediawiki/php-master/extensions/Flow/includes/Api/ApiFlow.php(106): Flow\Api\ApiFlowBaseGet->execute()
#12 /srv/mediawiki/php-master/includes/api/ApiMain.php(1435): Flow\Api\ApiFlow->execute()
#13 /srv/mediawiki/php-master/includes/api/ApiMain.php(508): ApiMain->executeAction()
#14 /srv/mediawiki/php-master/includes/api/ApiMain.php(479): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-master/api.php(83): ApiMain->execute()
#16 /srv/mediawiki/w/api.php(3): include(string)
#17 {main}"
code
:
"internal_api_error_DBQueryError"
info
:
"[V7N@SwpEEH8AAE1DMKMAAAAG] Database query error"
servedby
:
"deployment-mediawiki02"
Etonkovidova added subtasks: T133213: [betalabs-Regression] 'Browse topics' "[FLOW] Rendered topic not found when attempting to scroll", T139299: Cannot load an old Flow topic by clicking on its title in "Browse topics".Aug 16 2016, 9:05 PM
SBisson merged a task: T147017: Fatal database query error when scrolling up on Flow board.Sep 29 2016, 7:03 PM
SBisson added a subscriber: Krinkle.
Krinkle unsubscribed.Sep 29 2016, 7:37 PM
• jmatazzoni edited projects, added Collaboration-Team-Triage; removed Collab-Team-Q1-July-Sep-2016.Sep 29 2016, 11:06 PM
• jmatazzoni moved this task from Untriaged to Flow Holding Area on the Collaboration-Team-Triage board.
jcrespo added a comment.Feb 7 2017, 7:33 PM

I can still see this error on mediawiki.org, for example with url /w/api.php?action=flow&format=json&page=Extension_talk%3ALDAP_Authentication&submodule=view-topiclist&vtloffset-dir=rev&vtllimit=10&vtloffset=20161122183706&vtlsortby=updated

Unknown column 'workflow_last_update_timestamp' in 'order clause'

Catrope merged a task: T159371: flow bad sql.Mar 1 2017, 8:08 PM
Catrope added a subscriber: Matanya.
Catrope merged a task: T161040: Unknown column 'workflow_last_update_timestamp' in 'order clause'.Mar 21 2017, 9:15 PM
Catrope added subscribers: thcipriani, Reedy, • demon.
Reedy added a parent task: T160549: MW-1.29.0-wmf.17 deployment blockers.Mar 21 2017, 9:17 PM
Reedy added a comment.Mar 21 2017, 9:56 PM
In T161040#3119455, @demon wrote:

Full stracktrace:

#0 /srv/mediawiki/php-1.29.0-wmf.17/includes/libs/rdbms/database/Database.php(919): Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-1.29.0-wmf.17/includes/libs/rdbms/database/Database.php(1255): Database->query(string, string)
#2 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Data/Storage/BasicDbStorage.php(148): Database->select(string, string, array, string, array)
#3 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Data/Storage/BasicDbStorage.php(163): Flow\Data\Storage\BasicDbStorage->find(array, array)
#4 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Data/Storage/BasicDbStorage.php(180): Flow\Data\Storage\BasicDbStorage->fallbackFindMulti(array, array)
#5 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Data/ObjectLocator.php(103): Flow\Data\Storage\BasicDbStorage->findMulti(array, array)
#6 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Data/ObjectLocator.php(64): Flow\Data\ObjectLocator->findMulti(array, array)
#7 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Data/Pager/Pager.php(130): Flow\Data\ObjectLocator->find(array, array)
#8 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Block/TopicList.php(479): Flow\Data\Pager\Pager->getPage(Closure$Flow\Block\TopicListBlock::getPage;495966428)
#9 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Block/TopicList.php(266): Flow\Block\TopicListBlock->getPage(array)
#10 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Api/ApiFlowBaseGet.php(33): Flow\Block\TopicListBlock->renderApi(array)
#11 /srv/mediawiki/php-1.29.0-wmf.17/extensions/Flow/includes/Api/ApiFlow.php(98): Flow\Api\ApiFlowBaseGet->execute()
#12 /srv/mediawiki/php-1.29.0-wmf.17/includes/api/ApiMain.php(1544): Flow\Api\ApiFlow->execute()
#13 /srv/mediawiki/php-1.29.0-wmf.17/includes/api/ApiMain.php(545): ApiMain->executeAction()
#14 /srv/mediawiki/php-1.29.0-wmf.17/includes/api/ApiMain.php(516): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.29.0-wmf.17/api.php(83): ApiMain->execute()
#16 /srv/mediawiki/w/api.php(3): include(string)
#17 {main}
Catrope added a comment.Mar 21 2017, 10:48 PM

I can reproduce this locally, it looks like it happens whenever api.php?action=flow&submodule=view-topic is invoked with vtloffset-dir=rev.

Catrope added a comment.Mar 21 2017, 11:44 PM

This appears to have been caused by rEFLW699c5d86fcea: Fix history pagination and give user the number of entries they requested which made TopKIndex explicitly not match any reverse queries. This breaks a hack whereby order-by-timestamp queries are directed to a special instance of TopKIndex that uses a special storage backend (TopicListLastUpdatedStorage instead of TopicListStorage) which knows to do the join against the workflow table. This worked in forward mode, rEFLW699c5d86fcea: Fix history pagination and give user the number of entries they requested made it so that TopKIndex never matches reverse-ordered queries, so in reverse mode the query wouldn't match any of the indexes, which would throw a NoIndexException, which then gets caught and the query falls back to using the backend storage class directly, but because that is TopicListStorage rather than TopicListLastUpdatedStorage here, it doesn't know to do the join and explodes.

In other words, the code was trying to be too clever and only do the join in exactly the right circumstances, and that came crumbling down when Matt made a slight (and on its face sensible) change to how index matching works.

gerritbot subscribed.Mar 22 2017, 1:52 AM

Change 344073 had a related patch set uploaded (by Catrope):
[mediawiki/extensions/Flow] Make sure topiclist queries always join against workflow table

https://gerrit.wikimedia.org/r/344073

gerritbot added a project: Patch-For-Review.Mar 22 2017, 1:52 AM
gerritbot added a comment.Mar 22 2017, 6:58 PM

Change 344073 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Make sure topiclist queries always join against workflow table

https://gerrit.wikimedia.org/r/344073

ReleaseTaggerBot added a project: MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)).Mar 22 2017, 7:00 PM
gerritbot added a comment.Mar 22 2017, 7:31 PM

Change 344188 had a related patch set uploaded (by Catrope):
[mediawiki/extensions/Flow@wmf/1.29.0-wmf.17] Make sure topiclist queries always join against workflow table

https://gerrit.wikimedia.org/r/344188

gerritbot added a comment.Mar 22 2017, 7:45 PM

Change 344191 had a related patch set uploaded (by Catrope):
[mediawiki/extensions/Flow@master] Don't attempt to do reverse pagination in flow-loadmore

https://gerrit.wikimedia.org/r/344191

gerritbot added a comment.Mar 22 2017, 7:51 PM

Change 344188 merged by jenkins-bot:
[mediawiki/extensions/Flow@wmf/1.29.0-wmf.17] Make sure topiclist queries always join against workflow table

https://gerrit.wikimedia.org/r/344188

Stashbot subscribed.Mar 22 2017, 7:55 PM

Mentioned in SAL (#wikimedia-operations) [2017-03-22T19:55:44Z] <thcipriani@tin> Synchronized php-1.29.0-wmf.17/extensions/Flow: [[gerrit:344188|Make sure topiclist queries always join against workflow table]] T121644 (duration: 00m 59s)

ReleaseTaggerBot edited projects, added MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)); removed MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)).Mar 22 2017, 8:01 PM
gerritbot added a comment.Mar 22 2017, 8:37 PM

Change 344191 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Don't attempt to do reverse pagination in flow-loadmore

https://gerrit.wikimedia.org/r/344191

Catrope edited projects, added Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017); removed Collaboration-Team-Triage.Mar 22 2017, 8:56 PM
Catrope moved this task from Untriaged to QA Review on the Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017) board.
ReleaseTaggerBot edited projects, added MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)); removed MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)).Mar 22 2017, 9:00 PM
Etonkovidova added a comment.Mar 23 2017, 6:36 PM

Checked in betalabs following the description of steps for http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA . The issue seems to be fixed.

QA Recommendation: Resolve

Etonkovidova moved this task from QA Review to Product Review on the Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017) board.Mar 23 2017, 6:37 PM
• jmatazzoni closed this task as Resolved.Mar 23 2017, 8:28 PM
• jmatazzoni claimed this task.
Reedy removed a project: Patch-For-Review.May 17 2019, 9:46 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
Reedy mentioned this in T250542: Fix _createRevPagination in flow-board-loadmore.js.Apr 18 2020, 12:58 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL