Page MenuHomePhabricator

DBQuery Error: testwiki.growthexperiments_link_recommendations' doesn't exist
Closed, ResolvedPublic

Description

I have seen these errors showing up on logstash: https://logstash.wikimedia.org/goto/e0d362107ae2bf37b7523310ad08e70d

Error 1146 from GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationStore::getByRevId, Table 'testwiki.growthexperiments_link_recommendations' doesn't exist (10.64.32.82) SELECT  gelr_page,gelr_data  FROM `growthexperiments_link_recommendations`    WHERE gelr_revision = xxx  LIMIT 1   10.64.32.82

Stacktrace:

from /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1703)
#0 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1687): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1662): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1231): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(1913): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/database/Database.php(2013): Wikimedia\Rdbms\Database->select(string, array, array, string, array, array)
#5 /srv/mediawiki/php-1.36.0-wmf.33/extensions/GrowthExperiments/includes/NewcomerTasks/AddLink/LinkRecommendationStore.php(65): Wikimedia\Rdbms\Database->selectRow(string, array, array, string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.33/extensions/GrowthExperiments/includes/HomepageHooks.php(912): GrowthExperiments\NewcomerTasks\AddLink\LinkRecommendationStore->getByRevId(integer)
#7 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookContainer.php(163): GrowthExperiments\HomepageHooks->onSearchDataForIndex(array, JavaScriptContentHandler, WikiPage, ParserOutput, CirrusSearch\CirrusSearch)
#8 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookRunner.php(3320): MediaWiki\HookContainer\HookContainer->run(string, array)
#9 /srv/mediawiki/php-1.36.0-wmf.33/includes/content/ContentHandler.php(1405): MediaWiki\HookContainer\HookRunner->onSearchDataForIndex(array, JavaScriptContentHandler, WikiPage, ParserOutput, CirrusSearch\CirrusSearch)
#10 /srv/mediawiki/php-1.36.0-wmf.33/includes/content/TextContentHandler.php(157): ContentHandler->getDataForSearchIndex(WikiPage, ParserOutput, CirrusSearch\CirrusSearch)
#11 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CirrusSearch/includes/BuildDocument/ParserOutputPageProperties.php(88): TextContentHandler->getDataForSearchIndex(WikiPage, ParserOutput, CirrusSearch\CirrusSearch)
#12 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CirrusSearch/includes/BuildDocument/ParserOutputPageProperties.php(68): CirrusSearch\BuildDocument\ParserOutputPageProperties->finalizeReal(Elastica\Document, WikiPage, ParserCache, CirrusSearch\CirrusSearch)
#13 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CirrusSearch/includes/BuildDocument/BuildDocument.php(165): CirrusSearch\BuildDocument\ParserOutputPageProperties->finalize(Elastica\Document, Title)
#14 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CirrusSearch/includes/DataSender.php(299): CirrusSearch\BuildDocument\BuildDocument->finalize(Elastica\Document)
#15 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CirrusSearch/includes/Job/ElasticaWrite.php(136): CirrusSearch\DataSender->sendData(string, array)
#16 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CirrusSearch/includes/Job/JobTraits.php(136): CirrusSearch\Job\ElasticaWrite->doJob()
#17 /srv/mediawiki/php-1.36.0-wmf.33/extensions/EventBus/includes/JobExecutor.php(79): CirrusSearch\Job\CirrusGenericJob->run()
#18 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#19 {main}

Event Timeline

kostajh edited projects, added Growth-Team (Current Sprint); removed Growth-Team.
kostajh added a subscriber: kostajh.

Seems like we've missed setting a feature flag to prevent this query from occurring. @Tgr do you mind looking?

Change 667869 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] HomepageHooks: Block search data hook if link recommendations are off

https://gerrit.wikimedia.org/r/667869

kostajh moved this task from Incoming to Code Review on the Growth-Team (Current Sprint) board.

Seems like we've missed setting a feature flag to prevent this query from occurring. @Tgr do you mind looking?

Nevermind on that, I saw the problem and pushed a patch.

Ugh, sorry, that was stupid.

Caused by rEGREd9ca27e8e161: Update search index after a recommendation has been submitted which was merged on Feb 24, so riding this week's train.

Change 667869 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] HomepageHooks: Block search data hook if link recommendations are off

https://gerrit.wikimedia.org/r/667869

Change 667811 had a related patch set uploaded (by Gergő Tisza; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.33] HomepageHooks: Block search data hook if link recommendations are off

https://gerrit.wikimedia.org/r/667811

Change 667811 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.33] HomepageHooks: Block search data hook if link recommendations are off

https://gerrit.wikimedia.org/r/667811

Mentioned in SAL (#wikimedia-operations) [2021-03-02T15:35:22Z] <tgr@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/GrowthExperiments/: Backport: [[gerrit:667811|HomepageHooks: Block search data hook if link recommendations are off (T276224)]] (duration: 01m 13s)

Backported, don't see the errors in the log anymore. Thanks for catching @Marostegui and @kostajh!

Reviewed other uses of the link recommendation store; it's called in the two REST handlers, which abort with a 404 if link recommendations are disabled, and in HomepageHooks, which sets an error message which is eventually output via mw.log.error. So I think we are good here.

Thanks for getting this fixed so fast!