Page MenuHomePhabricator

Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
Closed, ResolvedPublic

Description

Looking at our DB-related kibana errors I've noticed a background noise of this recurring error:

Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.

Those are the first affected websites (count of errors) in the last 2 days:

WebsiteCount
commons.wikimedia.org9588
www.mediawiki.org6996
fr.wikisource.org5485
he.wikipedia.org5023
fr.wiktionary.org2434
ca.wikipedia.org2111
bg.wiktionary.org1985
ru.wiktionary.org1898
pl.wikisource.org1567
jobrunner.discovery.wmnet1176
it.wikisource.org1168
pt.wikinews.org1142
kn.wikibooks.org960

Example of full logged error:
https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.03.30/mediawiki?id=AWJ3luG7U5mXXxqDcR1H&_g=()

Stack trace:

#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(184): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(188): Closure$SpecialWhatLinksHere::showIndirectLinks()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(292): SpecialWhatLinksHere->showIndirectLinks()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(98): SpecialWhatLinksHere->showIndirectLinks()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): SpecialWhatLinksHere->execute()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#11 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#12 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#13 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#14 /srv/mediawiki/w/index.php(3): include()
#15 {main}

Event Timeline

Volans created this task.Mar 30 2018, 4:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2018, 4:15 PM
jcrespo set Security to Software security bug.Apr 2 2018, 9:19 AM
jcrespo added a project: Security.
jcrespo changed the visibility from "Public (No Login Required)" to "Custom Policy".
jcrespo added a subscriber: jcrespo.

Potential SQL injection

jcrespo triaged this task as High priority.Apr 2 2018, 9:22 AM
jcrespo moved this task from Triage to Next on the DBA board.
jcrespo added subscribers: Bawolff, Anomie.

Another malformed query potentially dangerous?

First occurence: 2018-03-28T01:14:55

It seems related to SpecialWhatLinksHere

jcrespo assigned this task to aaron.Apr 2 2018, 9:39 AM
Anomie added a comment.Apr 2 2018, 1:59 PM

The warning was added in rMWd395dfb039fc: rdbms: make selectRowCount() use $var argument to exclude NULLs as a deprecation warning.

rMWcc9a66e91bbc: Make SpecialWhatLinkshere::showIndirectLinks use buildSelectSubquery() is fixing one deprecated usage, not causing it.

I don't see anything here that indicates an SQL injection or a need to block the train.

Ok, then lowering priority.

jcrespo removed aaron as the assignee of this task.Apr 2 2018, 2:05 PM
jcrespo lowered the priority of this task from High to Medium.
jcrespo removed projects: Performance-Team, Security.
jcrespo changed the visibility from "Custom Policy" to "Public (No Login Required)".
Restricted Application added a project: Security. · View Herald TranscriptApr 2 2018, 2:06 PM

We can even close it as invalid if you think that is appropiate.

Anomie added a comment.Apr 2 2018, 2:12 PM

The deprecated uses still need cleaning up, so it could be kept open for that purpose.

Anomie added a comment.Apr 2 2018, 2:17 PM

Looking through the different stack traces associated with this message since 2018-04-01 12:57:04, I see only two causes, which both seem to have already been fixed in master by rMWcc9a66e91bbc: Make SpecialWhatLinkshere::showIndirectLinks use buildSelectSubquery() and rEANDd801bd3f6017: Use modern form of table alias usage in select() calls.

In the last 12h this has caused almost 1.5M errors: https://logstash.wikimedia.org/goto/e2a6f0cc6b65ccb88081e09216d1c7d7

trace	       	#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(184): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(192): Closure$SpecialWhatLinksHere::showIndirectLinks()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(98): SpecialWhatLinksHere->showIndirectLinks()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): SpecialWhatLinksHere->execute()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#11 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#12 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#13 /srv/mediawiki/w/index.php(3): include()
#14 {main}
Anomie added a comment.Apr 3 2018, 1:36 PM

In the last 12h this has caused almost 1.5M errors: https://logstash.wikimedia.org/goto/e2a6f0cc6b65ccb88081e09216d1c7d7

Since it sounds like the warning volume is problematic, I'll backport those two fixes to wmf.27 now instead of letting them ride the train.

Indeed - we relay a lot on that dashboard for our daily work, specially when doing changes. So having that huge amount of errors per minute can make us easily miss a more important or new error.
Thanks for taking the time to backport those.

Mentioned in SAL (#wikimedia-operations) [2018-04-03T13:47:02Z] <anomie@tin> Synchronized php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php: Backporting fix for T191116 ([[gerrit:423688]]) (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-04-03T13:48:22Z] <anomie@tin> Synchronized php-1.31.0-wmf.27/extensions/intersection/DynamicPageList.hooks.php: Backporting fix for T191116 ([[gerrit:423689]]) (duration: 00m 58s)

I can already see the decrease: https://logstash.wikimedia.org/goto/6cecb4a7d37409e0a5a622cf1dc3895f

I have agreed with @Anomie to monitor this for a few more hours and if no more errors show up, we will close the task by tomorrow.
Thanks for backporting it!

There are still some errors: https://logstash.wikimedia.org/goto/da09ff7550028cd215ca039f9ebfecc2
A lot less than before, but still some (519 in the last 12h)

t  message	       	Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
t  mwversion	       	1.31.0-wmf.27
t  normalized_message	       	Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
t  server	       	fr.wikinews.org
t  shard	       	s3
t  tags	       	syslog, es, es
t  trace	       	#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php(356): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php(96): GoogleNewsSitemap->getCategories()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): GoogleNewsSitemap->execute()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#11 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#12 /srv/mediawiki/w/index.php(3): include()
#13 {main}
jcrespo moved this task from Next to Blocked external/Not db team on the DBA board.Apr 4 2018, 2:21 PM
Anomie added a comment.Apr 4 2018, 4:16 PM

Looks like Aaron already fixed that one too: rEGNS2627206c4fb2: Use modern form of table alias usage in select() calls. Let me know if you'd like that backported too, otherwise it should go away once the train finishes tomorrow.

We can wait for it to be gone tomorrow :)
Thanks!

Marostegui closed this task as Resolved.Apr 5 2018, 5:47 AM
Marostegui assigned this task to Anomie.

And this is gone now after the train: https://logstash.wikimedia.org/goto/da09ff7550028cd215ca039f9ebfecc2
Thanks!

saper added a subscriber: saper.Aug 11 2018, 7:18 PM

Just ran into this issue when troubleshooting something on 1.31.0, pretty confusing!

Reedy added a subscriber: Reedy.Aug 11 2018, 7:26 PM

Just ran into this issue when troubleshooting something on 1.31.0, pretty confusing!

Might need some more information as to where/from what queries. A quick look at the commits linked above, they're all in 1.31 already

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM