Page MenuHomePhabricator

SecurePoll CreatePage can no longer correctly select "remote" wiki databases that aren't in the same cluster
Closed, ResolvedPublicPRODUCTION ERROR

Description

Caused by T276076: Port CreatePage to have service dependencies / rESPO71a5593dc9b4: Port CreatePage to have service dependencies

Similar/related errors will occur on other SecurePoll pages due to

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1049: Unknown database 'altwiki' (db2105)
Function: Wikimedia\Rdbms\DatabaseMysqlBase::doSelectDomain
Query: USE `altwiki`
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/database/Database.php(1794)
#0 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/database/Database.php(1778): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/database/Database.php(1753): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/database/DatabaseMysqlBase.php(211): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string)
#3 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/database/Database.php(2874): Wikimedia\Rdbms\DatabaseMysqlBase->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#4 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1269): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#5 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(988): Wikimedia\Rdbms\LoadBalancer->getForeignConnection(integer, string, integer)
#6 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(964): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#7 /srv/mediawiki/php-1.37.0-wmf.16/extensions/SecurePoll/includes/Pages/CreatePage.php(597): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string)
#8 /srv/mediawiki/php-1.37.0-wmf.16/includes/htmlform/HTMLForm.php(719): MediaWiki\Extensions\SecurePoll\Pages\CreatePage->processInput(array, OOUIHTMLForm)
#9 /srv/mediawiki/php-1.37.0-wmf.16/includes/htmlform/HTMLForm.php(609): HTMLForm->trySubmit()
#10 /srv/mediawiki/php-1.37.0-wmf.16/extensions/SecurePoll/includes/Pages/CreatePage.php(507): HTMLForm->tryAuthorizedSubmit()
#11 /srv/mediawiki/php-1.37.0-wmf.16/extensions/SecurePoll/includes/SpecialSecurePoll.php(70): MediaWiki\Extensions\SecurePoll\Pages\CreatePage->execute(array)
#12 /srv/mediawiki/php-1.37.0-wmf.16/includes/specialpage/SpecialPage.php(646): MediaWiki\Extensions\SecurePoll\SpecialSecurePoll->execute(string)
#13 /srv/mediawiki/php-1.37.0-wmf.16/includes/specialpage/SpecialPageFactory.php(1365): SpecialPage->run(string)
#14 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(314): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#15 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(925): MediaWiki->performRequest()
#16 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(559): MediaWiki->main()
#17 /srv/mediawiki/php-1.37.0-wmf.16/index.php(53): MediaWiki->run()
#18 /srv/mediawiki/php-1.37.0-wmf.16/index.php(46): wfIndexMain()
#19 /srv/mediawiki/w/index.php(3): require(string)
#20 {main}
Impact
Notes

Event Timeline

Reedy triaged this task as High priority.Jul 30 2021, 6:09 PM
Reedy updated the task description. (Show Details)

Change 709089 had a related patch set uploaded (by Majavah; author: Majavah):

[mediawiki/extensions/SecurePoll@master] Use correct load balancers for remote databases

https://gerrit.wikimedia.org/r/709089

Change 708993 had a related patch set uploaded (by Reedy; author: Majavah):

[mediawiki/extensions/SecurePoll@REL1_36] Use correct load balancers for remote databases

https://gerrit.wikimedia.org/r/708993

Change 708994 had a related patch set uploaded (by Reedy; author: Majavah):

[mediawiki/extensions/SecurePoll@wmf/1.37.0-wmf.16] Use correct load balancers for remote databases

https://gerrit.wikimedia.org/r/708994

Change 709089 merged by jenkins-bot:

[mediawiki/extensions/SecurePoll@master] Use correct load balancers for remote databases

https://gerrit.wikimedia.org/r/709089

I'll see about getting the .16 patch deployed over the weekend....

Change 708993 merged by jenkins-bot:

[mediawiki/extensions/SecurePoll@REL1_36] Use correct load balancers for remote databases

https://gerrit.wikimedia.org/r/708993

Change 708994 merged by jenkins-bot:

[mediawiki/extensions/SecurePoll@wmf/1.37.0-wmf.16] Use correct load balancers for remote databases

https://gerrit.wikimedia.org/r/708994

Mentioned in SAL (#wikimedia-operations) [2021-07-31T12:40:00Z] <reedy@deploy1002> Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 T287782 (duration: 00m 58s)

Leaving open for now until fix is tested in WMF prod

Leaving open for now until fix is tested in WMF prod

@Majavah @Reedy I have tried to create a new "remote" (i.e. selecting "All wikis" option in the "For wiki" dropdown) poll on votewiki and testwiki, both return the error:

(Cannot access the database: Cannot access the database: Connection timed out (clouddb2001-dev) (clouddb2001-dev))

Backtrace:

#0 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(992): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(964): Wikimedia\Rdbms\LoadBalancer->getServerConnection(0, 'labtestwiki', 4)
#2 /srv/mediawiki/php-1.37.0-wmf.16/extensions/SecurePoll/includes/Pages/CreatePage.php(599): Wikimedia\Rdbms\LoadBalancer->getConnection(-2, Array, 'labtestwiki')
#3 /srv/mediawiki/php-1.37.0-wmf.16/includes/htmlform/HTMLForm.php(719): MediaWiki\Extensions\SecurePoll\Pages\CreatePage->processInput(Array, Object(OOUIHTMLForm))
#4 /srv/mediawiki/php-1.37.0-wmf.16/includes/htmlform/HTMLForm.php(609): HTMLForm->trySubmit()
#5 /srv/mediawiki/php-1.37.0-wmf.16/extensions/SecurePoll/includes/Pages/CreatePage.php(508): HTMLForm->tryAuthorizedSubmit()
#6 /srv/mediawiki/php-1.37.0-wmf.16/extensions/SecurePoll/includes/SpecialSecurePoll.php(70): MediaWiki\Extensions\SecurePoll\Pages\CreatePage->execute(Array)
#7 /srv/mediawiki/php-1.37.0-wmf.16/includes/specialpage/SpecialPage.php(646): MediaWiki\Extensions\SecurePoll\SpecialSecurePoll->execute('create')
#8 /srv/mediawiki/php-1.37.0-wmf.16/includes/specialpage/SpecialPageFactory.php(1365): SpecialPage->run('create')
#9 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(314): MediaWiki\SpecialPage\SpecialPageFactory->executePath('SecurePoll/crea...', Object(RequestContext))
#10 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(925): MediaWiki->performRequest()
#11 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(559): MediaWiki->main()
#12 /srv/mediawiki/php-1.37.0-wmf.16/index.php(53): MediaWiki->run()
#13 /srv/mediawiki/php-1.37.0-wmf.16/index.php(46): wfIndexMain()
#14 /srv/mediawiki/w/index.php(3): require('/srv/mediawiki/...')
#15 {main}

It is possible, however, to create an election which is "local" (i.e. selecting the "This wiki" option in the "For wiki" dropdown).

Lol. It’s trying to connect to Wikitech, which it shouldn’t

Progress in the right direction though

I’m guessing wikitech/labawiki has been put into some list that securepoll is using… since it was last tested

Will have a look later :)! Shouldn’t be a difficult fix

Thanks for testing it!

(Cannot access the database: Cannot access the database: Connection timed out (clouddb2001-dev) (clouddb2001-dev))

So it's trying to access the database of labtestwikitech... fun. Looking at Puppet the database has the standard core mediawiki grants but it's firewalled so that normal appservers (like in this case) can't access it. SecurePoll is disabled on those wikis, but as far as I can see securepoll has always been accessing all the databases known to $wgConf. So I guess we need to somehow tell securepoll which databases to (not) use.

We can always put a quick hack in to unblock the elections

Change 709247 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/SecurePoll@master] Skip 'labswiki' and 'labtestwiki' in FormStore::getWikiList()

https://gerrit.wikimedia.org/r/709247

Following on from @Reedy's inline comment on the patch above:

$wgSecurePollCreateWikiGroups is set to [ 'securepollglobal' => 'securepoll-dblist-securepollglobal' ] for all wikis. AIUI that should mean that there's an entry in the For wiki dropdown on Special:SecurePoll/create with a label corresponding to the securepoll-dblist-securepollglobal message.

On votewiki, I'd expect to see an entry with the label "Board Election" (per https://vote.wikimedia.org/wiki/MediaWiki:Securepoll-dblist-securepollglobal) but I don't:

Screenshot 2021-08-02 at 11.12.25.png (154×713 px, 10 KB)

Edit

… presumably because $wgSecurePollCreateWikiGroupDir isn't set in production.

Change 709450 had a related patch set uploaded (by Reedy; author: Zabe):

[mediawiki/extensions/SecurePoll@wmf/1.36.0-wmf.17] Skip wikis without SecurePoll in FormStore::getWikiList()

https://gerrit.wikimedia.org/r/709450

Change 709247 merged by jenkins-bot:

[mediawiki/extensions/SecurePoll@master] Skip wikis without SecurePoll in FormStore::getWikiList()

https://gerrit.wikimedia.org/r/709247

Change 709450 abandoned by Reedy:

[mediawiki/extensions/SecurePoll@wmf/1.36.0-wmf.17] Skip wikis without SecurePoll in FormStore::getWikiList()

Reason:

https://gerrit.wikimedia.org/r/709450

Change 709451 had a related patch set uploaded (by Reedy; author: Zabe):

[mediawiki/extensions/SecurePoll@wmf/1.37.0-wmf.16] Skip wikis without SecurePoll in FormStore::getWikiList()

https://gerrit.wikimedia.org/r/709451

Change 709451 merged by jenkins-bot:

[mediawiki/extensions/SecurePoll@wmf/1.37.0-wmf.16] Skip wikis without SecurePoll in FormStore::getWikiList()

https://gerrit.wikimedia.org/r/709451

Mentioned in SAL (#wikimedia-operations) [2021-08-02T12:08:44Z] <reedy@deploy1002> Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 (duration: 00m 57s)

@Reedy @Zabe @Majavah Trying to create an election with "All wiki" selected on votewiki and testwiki, get some sort of timeout:

[19e3148a-a7d9-4c40-9aa2-bf5e49d774e1] /wiki/Special:SecurePoll/create Wikimedia\Rdbms\DBTransactionSizeError: Transaction spent 4.2640490531921 second(s) in writes, exceeding the limit of 3

Backtrace:

from /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1748)
#0 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2265): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#1 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1762): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#2 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactory.php(249): Wikimedia\Rdbms\LoadBalancer->approveMasterChanges(array, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(236): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#4 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactory.php(251): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure, array)
#5 /srv/mediawiki/php-1.37.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactory.php(310): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#6 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(684): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string, array)
#7 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(654): MediaWiki::preOutputCommit(RequestContext, Closure)
#8 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(948): MediaWiki->doPreOutputCommit(Closure)
#9 /srv/mediawiki/php-1.37.0-wmf.16/includes/MediaWiki.php(559): MediaWiki->main()
#10 /srv/mediawiki/php-1.37.0-wmf.16/index.php(53): MediaWiki->run()
#11 /srv/mediawiki/php-1.37.0-wmf.16/index.php(46): wfIndexMain()
#12 /srv/mediawiki/w/index.php(3): require(string)
#13 {main}

I had the same error, but on a second try I was able to create https://test.wikipedia.org/wiki/SecurePoll:828.

While it's an error in the same process, it's a different issue from the original. And as it will work sometimes, it's clear the original issue is fixed.

It's possible that it may require some longer timeout on this process, as it could take a while to do all those writes having to deal with many connections...

I had the same error, but on a second try I was able to create https://test.wikipedia.org/wiki/SecurePoll:828.

Oh. I tried 4 times in a row (on votewiki) but was not able to :(

Change 709589 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/SecurePoll@master] Use real transactions when creating an election

https://gerrit.wikimedia.org/r/709589

tstarling subscribed.

Change 709589 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

Sorry, linked the wrong bug.