Page MenuHomePhabricator

Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
Closed, ResolvedPublicPRODUCTION ERROR

Description

Looking at our DB-related kibana errors I've noticed a background noise of this recurring error:

Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.

Those are the first affected websites (count of errors) in the last 2 days:

WebsiteCount
commons.wikimedia.org9588
www.mediawiki.org6996
fr.wikisource.org5485
he.wikipedia.org5023
fr.wiktionary.org2434
ca.wikipedia.org2111
bg.wiktionary.org1985
ru.wiktionary.org1898
pl.wikisource.org1567
jobrunner.discovery.wmnet1176
it.wikisource.org1168
pt.wikinews.org1142
kn.wikibooks.org960

Example of full logged error:
https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.03.30/mediawiki?id=AWJ3luG7U5mXXxqDcR1H&_g=()

Stack trace:

#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(184): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(188): Closure$SpecialWhatLinksHere::showIndirectLinks()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(292): SpecialWhatLinksHere->showIndirectLinks()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(98): SpecialWhatLinksHere->showIndirectLinks()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): SpecialWhatLinksHere->execute()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#11 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#12 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#13 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#14 /srv/mediawiki/w/index.php(3): include()
#15 {main}

Event Timeline

jcrespo set Security to Software security bug.Apr 2 2018, 9:19 AM
jcrespo added a project: acl*security.
jcrespo changed the visibility from "Public (No Login Required)" to "Custom Policy".
jcrespo added a subscriber: jcrespo.

Potential SQL injection

jcrespo triaged this task as High priority.Apr 2 2018, 9:22 AM
jcrespo moved this task from Triage to Pending comment on the DBA board.
jcrespo added subscribers: Bawolff, Anomie.

Another malformed query potentially dangerous?

First occurence: 2018-03-28T01:14:55

It seems related to SpecialWhatLinksHere

The warning was added in rMWd395dfb039fc: rdbms: make selectRowCount() use $var argument to exclude NULLs as a deprecation warning.

rMWcc9a66e91bbc: Make SpecialWhatLinkshere::showIndirectLinks use buildSelectSubquery() is fixing one deprecated usage, not causing it.

I don't see anything here that indicates an SQL injection or a need to block the train.

jcrespo removed aaron as the assignee of this task.Apr 2 2018, 2:05 PM
jcrespo lowered the priority of this task from High to Medium.
jcrespo changed the visibility from "Custom Policy" to "Public (No Login Required)".

We can even close it as invalid if you think that is appropiate.

The deprecated uses still need cleaning up, so it could be kept open for that purpose.

Looking through the different stack traces associated with this message since 2018-04-01 12:57:04, I see only two causes, which both seem to have already been fixed in master by rMWcc9a66e91bbc: Make SpecialWhatLinkshere::showIndirectLinks use buildSelectSubquery() and rEANDd801bd3f6017: Use modern form of table alias usage in select() calls.

In the last 12h this has caused almost 1.5M errors: https://logstash.wikimedia.org/goto/e2a6f0cc6b65ccb88081e09216d1c7d7

trace	       	#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(184): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(192): Closure$SpecialWhatLinksHere::showIndirectLinks()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(98): SpecialWhatLinksHere->showIndirectLinks()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): SpecialWhatLinksHere->execute()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#11 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#12 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#13 /srv/mediawiki/w/index.php(3): include()
#14 {main}

In the last 12h this has caused almost 1.5M errors: https://logstash.wikimedia.org/goto/e2a6f0cc6b65ccb88081e09216d1c7d7

Since it sounds like the warning volume is problematic, I'll backport those two fixes to wmf.27 now instead of letting them ride the train.

Indeed - we relay a lot on that dashboard for our daily work, specially when doing changes. So having that huge amount of errors per minute can make us easily miss a more important or new error.
Thanks for taking the time to backport those.

Mentioned in SAL (#wikimedia-operations) [2018-04-03T13:47:02Z] <anomie@tin> Synchronized php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php: Backporting fix for T191116 ([[gerrit:423688]]) (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-04-03T13:48:22Z] <anomie@tin> Synchronized php-1.31.0-wmf.27/extensions/intersection/DynamicPageList.hooks.php: Backporting fix for T191116 ([[gerrit:423689]]) (duration: 00m 58s)

I can already see the decrease: https://logstash.wikimedia.org/goto/6cecb4a7d37409e0a5a622cf1dc3895f

I have agreed with @Anomie to monitor this for a few more hours and if no more errors show up, we will close the task by tomorrow.
Thanks for backporting it!

There are still some errors: https://logstash.wikimedia.org/goto/da09ff7550028cd215ca039f9ebfecc2
A lot less than before, but still some (519 in the last 12h)

t  message	       	Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
t  mwversion	       	1.31.0-wmf.27
t  normalized_message	       	Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
t  server	       	fr.wikinews.org
t  shard	       	s3
t  tags	       	syslog, es, es
t  trace	       	#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php(356): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php(96): GoogleNewsSitemap->getCategories()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): GoogleNewsSitemap->execute()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#11 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#12 /srv/mediawiki/w/index.php(3): include()
#13 {main}

Looks like Aaron already fixed that one too: rEGNS2627206c4fb2: Use modern form of table alias usage in select() calls. Let me know if you'd like that backported too, otherwise it should go away once the train finishes tomorrow.

We can wait for it to be gone tomorrow :)
Thanks!

Just ran into this issue when troubleshooting something on 1.31.0, pretty confusing!

Just ran into this issue when troubleshooting something on 1.31.0, pretty confusing!

Might need some more information as to where/from what queries. A quick look at the commits linked above, they're all in 1.31 already

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM

I noticed a flood of these for SpecialRecentChangesLinked in the DBQuery channel, https://logstash.wikimedia.org/goto/d65d99ed2ca4de02a45510954d44b338 in the last 2 months.

That's a different issue that got fixed.

Looks like https://gerrit.wikimedia.org/r/c/mediawiki/core/+/831082 was the fix.

Can this warning become an exception now (or the check removed completely)?