Cannot access the database: Unknown error (10.64.0.164)
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	brennen
	Mar 10 2021, 7:33 PM

Description

Error

MediaWiki version: 1.36.0-wmf.33

message

Cannot access the database: Unknown error (10.64.0.164)

exception.trace

from /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1507)
#0 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(995): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(960): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#2 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Echo/includes/EchoDbFactory.php(90): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array)
#3 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Echo/includes/model/Event.php(127): MWEchoDbFactory->getEchoDb(integer)
#4 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Echo/includes/EchoHooks.php(831): EchoEvent::create(array)
#5 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookContainer.php(333): EchoHooks::onLinksUpdateAfterInsert(LinksUpdate, string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookContainer.php(140): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#7 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookRunner.php(2393): MediaWiki\HookContainer\HookContainer->run(string, array)
#8 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/LinksUpdate.php(518): MediaWiki\HookContainer\HookRunner->onLinksUpdateAfterInsert(LinksUpdate, string, array)
#9 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/LinksUpdate.php(237): LinksUpdate->incrTableUpdate(string, string, array, array)
#10 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/LinksUpdate.php(194): LinksUpdate->doIncrementalUpdate()
#11 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/DeferredUpdates.php(513): LinksUpdate->doUpdate()
#12 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/RefreshSecondaryDataUpdate.php(101): DeferredUpdates::attemptUpdate(LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#13 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/DeferredUpdates.php(513): RefreshSecondaryDataUpdate->doUpdate()
#14 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/DeferredUpdates.php(390): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#15 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#16 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/DeferredUpdatesScope.php(264): DeferredUpdates::{closure}(RefreshSecondaryDataUpdate, integer)
#17 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#18 /srv/mediawiki/php-1.36.0-wmf.33/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#19 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(1097): DeferredUpdates::doUpdates(string)
#20 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(834): MediaWiki->restInPeace()
#21 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(846): MediaWiki->{closure}()
#22 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(584): MediaWiki->doPostOutputShutdown()
#23 /srv/mediawiki/php-1.36.0-wmf.33/index.php(53): MediaWiki->run()
#24 /srv/mediawiki/php-1.36.0-wmf.33/index.php(46): wfIndexMain()
#25 /srv/mediawiki/w/index.php(3): require(string)
#26 {main}

Impact

Unclear.

Notes

4 of these in last 4 hours, all from the EchoDB code path.

These surfacing in exception channel seems unusual?

Details

Request ID: YEkUMT3TKH5GsdsqARElBQAAAMU
Request URL: https://el.wikipedia.org/w/index.php?title=[title]&action=submit

Related Objects

Mentioned Here: T268715: Occasional "Cannot access the database: Unknown error" in Wikimedia production

Event Timeline

brennen created this task.Mar 10 2021, 7:33 PM

Restricted Application added a project: Growth-Team. · View Herald TranscriptMar 10 2021, 7:33 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Krinkle moved this task from Untriaged to Mar 2021 on the Wikimedia-production-error board.Mar 10 2021, 7:37 PM

Krinkle updated the task description. (Show Details)

Krinkle edited Stack Trace. (Show Details)

See also T268715 for the same error text

That particular host (10.64.0.164) is db1103, which is x1 master.
Those errors in general aren't new, as pointed out above.
In this specific case, without having a more specific error there's not much we can debug apart from taking a look at the graphs: https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1103&var-port=9104&from=now-24h&to=now
It does show a bit of an spike on writes, but nothing to crazy or out of the ordinary from what I can see.

There's not much else we can do at this point unless we get a way to see if there's a more specific error we can trace - I would be inclined to close this as a duplicate of T268715

• Marostegui edited projects, added Data-Persistence (work done); removed DBA.Mar 10 2021, 8:08 PM

LSobanski subscribed.Mar 11 2021, 10:20 AM

• mmodell closed this task as a duplicate of T268715: Occasional "Cannot access the database: Unknown error" in Wikimedia production.Apr 1 2021, 7:10 PM

Cannot access the database: Unknown error (10.64.0.164)Closed, DuplicatePublicPRODUCTION ERRORActions