Page MenuHomePhabricator
Create Task
Maniphest T191116

Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Looking at our DB-related kibana errors I've noticed a background noise of this recurring error:

Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.

Those are the first affected websites (count of errors) in the last 2 days:

WebsiteCount
commons.wikimedia.org9588
www.mediawiki.org6996
fr.wikisource.org5485
he.wikipedia.org5023
fr.wiktionary.org2434
ca.wikipedia.org2111
bg.wiktionary.org1985
ru.wiktionary.org1898
pl.wikisource.org1567
jobrunner.discovery.wmnet1176
it.wikisource.org1168
pt.wikinews.org1142
kn.wikibooks.org960

Example of full logged error:
https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.03.30/mediawiki?id=AWJ3luG7U5mXXxqDcR1H&_g=()

Stack trace:

#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(184): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(188): Closure$SpecialWhatLinksHere::showIndirectLinks()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(292): SpecialWhatLinksHere->showIndirectLinks()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(98): SpecialWhatLinksHere->showIndirectLinks()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): SpecialWhatLinksHere->execute()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#11 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#12 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#13 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#14 /srv/mediawiki/w/index.php(3): include()
#15 {main}

Related Objects

Event Timeline

Volans created this task.Mar 30 2018, 4:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2018, 4:15 PM
Marostegui added a project: MediaWiki-libs-Rdbms.Mar 30 2018, 5:37 PM
jcrespo set Security to Software security bug.Apr 2 2018, 9:19 AM
jcrespo added a project: acl*security.
jcrespo changed the visibility from "Public (No Login Required)" to "Custom Policy".
jcrespo subscribed.

Potential SQL injection

jcrespo triaged this task as High priority.Apr 2 2018, 9:22 AM
jcrespo moved this task from Triage to Pending comment on the DBA board.
jcrespo edited projects, added: MediaWiki-Action-API, MediaWiki-Platform-Team-Archived, Wikimedia-production-error; removed: MediaWiki-General.
jcrespo added subscribers: Bawolff, Anomie.

Another malformed query potentially dangerous?

jcrespo added a comment.Apr 2 2018, 9:30 AM

First occurence: 2018-03-28T01:14:55

jcrespo mentioned this in T183966: 1.31.0-wmf.27 deployment blockers.Apr 2 2018, 9:31 AM
jcrespo added a parent task: T183966: 1.31.0-wmf.27 deployment blockers.
jcrespo edited projects, added: MediaWiki-Special-pages; removed: MediaWiki-Action-API.Apr 2 2018, 9:36 AM

It seems related to SpecialWhatLinksHere

jcrespo edited projects, added: Performance-Team; removed: MediaWiki-Platform-Team-Archived.Apr 2 2018, 9:38 AM
jcrespo added a subscriber: aaron.

Probably cc9a66e91bbc44ab5b2d0159a467c9ac77a87bc1

jcrespo assigned this task to aaron.Apr 2 2018, 9:39 AM
Krinkle moved this task from Untriaged to Usage problem on the MediaWiki-libs-Rdbms board.Apr 2 2018, 12:57 PM
Krinkle moved this task from To triage to Special:WhatLinksHere on the MediaWiki-Special-pages board.
Anomie added a comment.Apr 2 2018, 1:59 PM

The warning was added in rMWd395dfb039fc: rdbms: make selectRowCount() use $var argument to exclude NULLs as a deprecation warning.

rMWcc9a66e91bbc: Make SpecialWhatLinkshere::showIndirectLinks use buildSelectSubquery() is fixing one deprecated usage, not causing it.

I don't see anything here that indicates an SQL injection or a need to block the train.

jcrespo added a comment.Apr 2 2018, 2:05 PM

Ok, then lowering priority.

jcrespo removed aaron as the assignee of this task.Apr 2 2018, 2:05 PM
jcrespo lowered the priority of this task from High to Medium.
jcrespo removed projects: Performance-Team, acl*security.
jcrespo changed the visibility from "Custom Policy" to "Public (No Login Required)".
Restricted Application added a project: acl*security. · View Herald TranscriptApr 2 2018, 2:06 PM
jcrespo removed a parent task: T183966: 1.31.0-wmf.27 deployment blockers.Apr 2 2018, 2:06 PM

We can even close it as invalid if you think that is appropiate.

Anomie added a comment.Apr 2 2018, 2:12 PM

The deprecated uses still need cleaning up, so it could be kept open for that purpose.

Anomie added a comment.Apr 2 2018, 2:17 PM

Looking through the different stack traces associated with this message since 2018-04-01 12:57:04, I see only two causes, which both seem to have already been fixed in master by rMWcc9a66e91bbc: Make SpecialWhatLinkshere::showIndirectLinks use buildSelectSubquery() and rEANDd801bd3f6017: Use modern form of table alias usage in select() calls.

Marostegui subscribed.Apr 3 2018, 5:26 AM

In the last 12h this has caused almost 1.5M errors: https://logstash.wikimedia.org/goto/e2a6f0cc6b65ccb88081e09216d1c7d7

trace	       	#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(184): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(192): Closure$SpecialWhatLinksHere::showIndirectLinks()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php(98): SpecialWhatLinksHere->showIndirectLinks()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): SpecialWhatLinksHere->execute()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#11 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#12 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#13 /srv/mediawiki/w/index.php(3): include()
#14 {main}
Anomie added a comment.Apr 3 2018, 1:36 PM
In T191116#4099497, @Marostegui wrote:

In the last 12h this has caused almost 1.5M errors: https://logstash.wikimedia.org/goto/e2a6f0cc6b65ccb88081e09216d1c7d7

Since it sounds like the warning volume is problematic, I'll backport those two fixes to wmf.27 now instead of letting them ride the train.

Marostegui added a comment.Apr 3 2018, 1:40 PM

Indeed - we relay a lot on that dashboard for our daily work, specially when doing changes. So having that huge amount of errors per minute can make us easily miss a more important or new error.
Thanks for taking the time to backport those.

Stashbot subscribed.Apr 3 2018, 1:47 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-03T13:47:02Z] <anomie@tin> Synchronized php-1.31.0-wmf.27/includes/specials/SpecialWhatlinkshere.php: Backporting fix for T191116 ([[gerrit:423688]]) (duration: 00m 58s)

Stashbot added a comment.Apr 3 2018, 1:48 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-03T13:48:22Z] <anomie@tin> Synchronized php-1.31.0-wmf.27/extensions/intersection/DynamicPageList.hooks.php: Backporting fix for T191116 ([[gerrit:423689]]) (duration: 00m 58s)

Marostegui added a comment.Apr 3 2018, 1:53 PM

I can already see the decrease: https://logstash.wikimedia.org/goto/6cecb4a7d37409e0a5a622cf1dc3895f

I have agreed with @Anomie to monitor this for a few more hours and if no more errors show up, we will close the task by tomorrow.
Thanks for backporting it!

Marostegui added a comment.Apr 4 2018, 5:31 AM

There are still some errors: https://logstash.wikimedia.org/goto/da09ff7550028cd215ca039f9ebfecc2
A lot less than before, but still some (519 in the last 12h)

t  message	       	Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
t  mwversion	       	1.31.0-wmf.27
t  normalized_message	       	Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way.
t  server	       	fr.wikinews.org
t  shard	       	s3
t  tags	       	syslog, es, es
t  trace	       	#0 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2200): Wikimedia\Rdbms\Database->tableName()
#1 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(2311): Wikimedia\Rdbms\Database->tableNameWithAlias()
#2 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1521): Wikimedia\Rdbms\Database->tableNamesWithIndexClauseOrJOIN()
#3 /srv/mediawiki/php-1.31.0-wmf.27/includes/libs/rdbms/database/Database.php(1495): Wikimedia\Rdbms\Database->selectSQLText()
#4 /srv/mediawiki/php-1.31.0-wmf.27/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php(356): Wikimedia\Rdbms\Database->select()
#5 /srv/mediawiki/php-1.31.0-wmf.27/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php(96): GoogleNewsSitemap->getCategories()
#6 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPage.php(522): GoogleNewsSitemap->execute()
#7 /srv/mediawiki/php-1.31.0-wmf.27/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run()
#8 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(288): SpecialPageFactory::executePath()
#9 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(861): MediaWiki->performRequest()
#10 /srv/mediawiki/php-1.31.0-wmf.27/includes/MediaWiki.php(524): MediaWiki->main()
#11 /srv/mediawiki/php-1.31.0-wmf.27/index.php(42): MediaWiki->run()
#12 /srv/mediawiki/w/index.php(3): include()
#13 {main}
jcrespo moved this task from Pending comment to Blocked external/Not db team on the DBA board.Apr 4 2018, 2:21 PM
Anomie added a comment.Apr 4 2018, 4:16 PM

Looks like Aaron already fixed that one too: rEGNS2627206c4fb2: Use modern form of table alias usage in select() calls. Let me know if you'd like that backported too, otherwise it should go away once the train finishes tomorrow.

Marostegui added a comment.Apr 4 2018, 4:18 PM

We can wait for it to be gone tomorrow :)
Thanks!

Marostegui closed this task as Resolved.Apr 5 2018, 5:47 AM
Marostegui assigned this task to Anomie.

And this is gone now after the train: https://logstash.wikimedia.org/goto/da09ff7550028cd215ca039f9ebfecc2
Thanks!

saper subscribed.Aug 11 2018, 7:18 PM

Just ran into this issue when troubleshooting something on 1.31.0, pretty confusing!

Reedy removed a project: acl*security.Aug 11 2018, 7:22 PM
Reedy subscribed.Aug 11 2018, 7:26 PM
In T191116#4496669, @saper wrote:

Just ran into this issue when troubleshooting something on 1.31.0, pretty confusing!

Might need some more information as to where/from what queries. A quick look at the commits linked above, they're all in 1.31 already

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM
aaron added a comment.Sep 28 2022, 4:35 PM

I noticed a flood of these for SpecialRecentChangesLinked in the DBQuery channel, https://logstash.wikimedia.org/goto/d65d99ed2ca4de02a45510954d44b338 in the last 2 months.

Restricted Application added a project: Performance-Team. · View Herald TranscriptSep 28 2022, 4:35 PM
Ladsgroup subscribed.Sep 28 2022, 4:39 PM

That's a different issue that got fixed.

aaron added a comment.Sep 28 2022, 4:46 PM

Looks like https://gerrit.wikimedia.org/r/c/mediawiki/core/+/831082 was the fix.

Can this warning become an exception now (or the check removed completely)?

Krinkle mentioned this in T318845: Disallow passing raw subqueries to IDatabase::tableName.Sep 28 2022, 5:49 PM
Krinkle subscribed.

Sounds like a good idea to me. I filed T318845: Disallow passing raw subqueries to IDatabase::tableName

Justin_C_Lloyd subscribed.Oct 18 2022, 6:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits