Wikitech: "Cannot access the database: Unknown error (10.64.32.72)"
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Jdforrester-WMF
	Dec 18 2019, 1:35 PM

Description

Error

MediaWiki version: 1.35.0-wmf.11

message

Cannot access the database: Unknown error (10.64.32.72)

Impact

Notes

Details

Request ID: 1023a5a1-9055-45ca-83e3-94f4f0f71551
Request URL: https://wikitech.wikimedia.org/w/api.php?uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&action=query&format=json&continue=&siprop=general%7Cnamespaces&meta=siteinfo%7Cuserinfo%7Cuserinfo

Stack Trace

exception.trace

#0 /srv/mediawiki/php-1.35.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(932): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.35.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(899): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#2 /srv/mediawiki/php-1.35.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1044): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#3 /srv/mediawiki/php-1.35.0-wmf.11/includes/GlobalFunctions.php(2580): Wikimedia\Rdbms\LoadBalancer->getMaintenanceConnectionRef(integer, array, string)
#4 /srv/mediawiki/php-1.35.0-wmf.11/includes/block/DatabaseBlock.php(257): wfGetDB(integer)
#5 /srv/mediawiki/php-1.35.0-wmf.11/includes/block/DatabaseBlock.php(1190): MediaWiki\Block\DatabaseBlock::newLoad(User, integer, boolean, string)
#6 /srv/mediawiki/php-1.35.0-wmf.11/includes/block/BlockManager.php(132): MediaWiki\Block\DatabaseBlock::newListFromTarget(User, string, boolean)
#7 /srv/mediawiki/php-1.35.0-wmf.11/includes/user/User.php(1799): MediaWiki\Block\BlockManager->getUserBlock(User, WebRequest, boolean)
#8 /srv/mediawiki/php-1.35.0-wmf.11/includes/user/User.php(2117): User->getBlockedStatus(boolean)
#9 /srv/mediawiki/php-1.35.0-wmf.11/includes/block/BlockManager.php(470): User->getBlock()
#10 /srv/mediawiki/php-1.35.0-wmf.11/includes/MediaWiki.php(751): MediaWiki\Block\BlockManager->trackBlockWithCookie(User, WebResponse)
#11 /srv/mediawiki/php-1.35.0-wmf.11/includes/api/ApiMain.php(565): MediaWiki::preOutputCommit(DerivativeContext)
#12 /srv/mediawiki/php-1.35.0-wmf.11/includes/api/ApiMain.php(510): ApiMain->executeActionWithErrorHandling()
#13 /srv/mediawiki/php-1.35.0-wmf.11/api.php(78): ApiMain->execute()
#14 /srv/mediawiki/w/api.php(3): require(string)
#15 {main}

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Revert "train: Rolling Wikitech back to 1.35.0-wmf.10"	operations/mediawiki-config	master	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	Jdforrester-WMF	T233859 1.35.0-wmf.11 deployment blockers
		Resolved	PRODUCTION ERROR	None	T241059 Wikitech: "Cannot access the database: Unknown error (10.64.32.72)"

Event Timeline

Jdforrester-WMF created this task.Dec 18 2019, 1:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 18 2019, 1:35 PM

Also seeing this on wmf.10 now that we've rolled back:

#0 /srv/mediawiki/php-1.35.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(932): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.35.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(899): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#2 /srv/mediawiki/php-1.35.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1044): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#3 /srv/mediawiki/php-1.35.0-wmf.10/includes/GlobalFunctions.php(2580): Wikimedia\Rdbms\LoadBalancer->getMaintenanceConnectionRef(integer, array, string)
#4 /srv/mediawiki/php-1.35.0-wmf.10/includes/resourceloader/ResourceLoaderModule.php(405): wfGetDB(integer)
#5 /srv/mediawiki/php-1.35.0-wmf.10/includes/resourceloader/ResourceLoaderFileModule.php(573): ResourceLoaderModule->getFileDependencies(ResourceLoaderContext)
#6 /srv/mediawiki/php-1.35.0-wmf.10/includes/resourceloader/ResourceLoaderFileModule.php(635): ResourceLoaderFileModule->getFileHashes(ResourceLoaderContext)
#7 /srv/mediawiki/php-1.35.0-wmf.10/includes/resourceloader/ResourceLoaderModule.php(819): ResourceLoaderFileModule->getDefinitionSummary(ResourceLoaderContext)
#8 /srv/mediawiki/php-1.35.0-wmf.10/includes/resourceloader/ResourceLoader.php(1084): ResourceLoaderModule->getVersionHash(ResourceLoaderContext)
#9 /srv/mediawiki/php-1.35.0-wmf.10/includes/resourceloader/ResourceLoader.php(790): ResourceLoader->makeModuleResponse(ResourceLoaderContext, array, array)
#10 /srv/mediawiki/php-1.35.0-wmf.10/load.php(43): ResourceLoader->respond(ResourceLoaderContext)
#11 /srv/mediawiki/w/load.php(3): require(string)

Jdforrester-WMF triaged this task as Unbreak Now! priority.Dec 18 2019, 1:37 PM

Jdforrester-WMF added a parent task: T233859: 1.35.0-wmf.11 deployment blockers.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptDec 18 2019, 1:38 PM

(Now the whole of Wikitech is down.)

Wikitech is back up after(?) the wikitech servers had php7.2-fpm restarted.

Change 559112 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] Revert "train: Rolling Wikitech back to 1.35.0-wmf.10"

https://gerrit.wikimedia.org/r/559112

Change 559112 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "train: Rolling Wikitech back to 1.35.0-wmf.10"

https://gerrit.wikimedia.org/r/559112

Mentioned in SAL (#wikimedia-operations) [2019-12-18T15:58:45Z] <jforrester@deploy1001> rebuilt and synchronized wikiversions files: train: Rolling Wikitech foward to 1.35.0-wmf.11 T233859 T241059

Having re-rolled the train, all seems quiet. Without replication steps, I'm going to mark this as Resolved.

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2019, 4:10 PM

Krinkle renamed this task from Cannot access the database: Unknown error (10.64.32.72) to Wikitech: "Cannot access the database: Unknown error (10.64.32.72)".Dec 18 2019, 5:40 PM

Wikitech: "Cannot access the database: Unknown error (10.64.32.72)"Closed, ResolvedPublicPRODUCTION ERRORActions