Fatal error: Call to fetch_object() on boolean (in DatabaseMysqli->mysqlFetchObject)
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	Krinkle
	Mar 17 2020, 4:21 PM

Description

Error

MediaWiki version: 1.35.0-wmf.23

message

Fatal error: Call to a member function fetch_object() on boolean

exception.trace

#0 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DatabaseMysqlBase.php(255): Wikimedia\Rdbms\DatabaseMysqli->mysqlFetchObject(boolean)
#1 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DatabaseMysqlBase.php(1091): Wikimedia\Rdbms\DatabaseMysqlBase->fetchObject(boolean)
#2 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2065): Wikimedia\Rdbms\DatabaseMysqlBase->serverIsReadOnly()
#3 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/objectcache/BagOStuff.php(148): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(integer)
#4 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2069): BagOStuff->getWithSetCallback(string, integer, Closure)
#5 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(966): Wikimedia\Rdbms\LoadBalancer->isMasterConnectionReadOnly(Wikimedia\Rdbms\DatabaseMysqli, integer)
#6 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(901): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#7 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1031): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#8 /srv/mediawiki/php-1.35.0-wmf.23/includes/externalstore/ExternalStoreDB.php(168): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string, integer)
#9 /srv/mediawiki/php-1.35.0-wmf.23/includes/externalstore/ExternalStoreDB.php(311): ExternalStoreDB->getReplica(string)
#10 /srv/mediawiki/php-1.35.0-wmf.23/includes/externalstore/ExternalStoreDB.php(66): ExternalStoreDB->fetchBlob(string, string, boolean)
#11 /srv/mediawiki/php-1.35.0-wmf.23/includes/externalstore/ExternalStoreAccess.php(52): ExternalStoreDB->fetchFromURL(string)
#12 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php(496): ExternalStoreAccess->fetchFromURL(string, array)
#13 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/objectcache/wancache/WANObjectCache.php(1471): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL, array)
#14 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/objectcache/wancache/WANObjectCache.php(1314): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#15 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php(500): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#16 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php(419): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#17 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php(286): MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, integer)
#18 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/objectcache/wancache/WANObjectCache.php(1471): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL, array)
#19 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/objectcache/wancache/WANObjectCache.php(1314): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#20 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php(291): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#21 /srv/mediawiki/php-1.35.0-wmf.23/maintenance/fetchText.php(81): MediaWiki\Storage\SqlBlobStore->getBlob(string)
#22 /srv/mediawiki/php-1.35.0-wmf.23/maintenance/doMaintenance.php(99): FetchText->execute()
#23 /srv/mediawiki/php-1.35.0-wmf.23/maintenance/fetchText.php(100): require_once(string)
#24 /srv/mediawiki/multiversion/MWScript.php(101): require_once(string)
#25 {main}

Impact

We see that some maintenance scripts have failed.

The code path that leads to the issue however does not seem to be specific to dumps or maintenance in any way. I would imagine it is a general bug in the Rdbms lib that could affect other web requests as well.

Notes

Details

Request ID: 63ce333480f33f0c6b979399
Request URL: n/a

	Subject	Repo	Branch	Lines +/-
	rdbms: Don't silence errors in DatabaseMysqlBase::serverIsReadOnly()	mediawiki/core	master	+1 -1

Customize query in gerrit

Event Timeline

Krinkle created this task.Mar 17 2020, 4:21 PM

Restricted Application added a project: Platform Engineering. · View Herald TranscriptMar 17 2020, 4:21 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

DatabaseMysqli.php

	/**
	 * @param mysqli_result $res
	 * @return stdClass|bool
	 */
	protected function mysqlFetchObject( $res ) {
		$object = $res->fetch_object();

DatabaseMysqlBase.php

	/**
	 * @param IResultWrapper|resource $res
	 */
	public function fetchObject( $res ) {
		$row = $this->mysqlFetchObject( ResultWrapper::unwrap( $res ) );

Then, we see that fetchObject is sometimes called conditionally after a query() and sometimes unconditionally.

DatabaseMysqlBase.php

	// unconditionally 
	public function serverIsReadOnly() {

		$flags = self::QUERY_IGNORE_DBO_TRX | self::QUERY_SILENCE_ERRORS;
		$res = $this->query( "SELECT @@GLOBAL.read_only AS Value", __METHOD__, $flags );
		$row = $this->fetchObject( $res );

	/// unconditionally

		$res = $this->query( 'SHOW SLAVE STATUS', __METHOD__, $flags );
		$row = $res ? $res->fetchObject() : false;

That looks suspicious but I am not familiar with the values that are being queried by serverIsReadOnly, nor the intended behaviour of QUERY_SILENCE_ERRORS to know whether it is supposed to be possible for query() to return false in this case.

Krinkle moved this task from Untriaged to Mar 2020 on the Wikimedia-production-error board.Mar 17 2020, 4:23 PM

It'd be interesting to know what the error is, that query doesn't seem like it should fail. The error should still have been logged somewhere, can you check?

QUERY_SILENCE_ERRORS is indeed intended to make query() return false instead of throw in certain cases. So either serverIsReadOnly() should check for false and do something sensible, or it should not pass QUERY_SILENCE_ERRORS and allow an exception to be thrown if the query somehow errors.

Anomie claimed this task.Mar 24 2020, 8:19 PM

Anomie edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.

Anomie moved this task from Inbox to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.

Anomie triaged this task as Medium priority.Mar 24 2020, 8:50 PM

Anomie moved this task from Ready (WIP:5) to Waiting for Review on the Platform Team Workboards (Clinic Duty Team) board.Mar 25 2020, 7:11 PM

Change 583416 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@master] rdbms: Don't silence errors in DatabaseMysqlBase::serverIsReadOnly()

https://gerrit.wikimedia.org/r/583416

gerritbot added a project: Patch-For-Review.Mar 25 2020, 7:11 PM

Change 583416 merged by jenkins-bot:
[mediawiki/core@master] rdbms: Don't silence errors in DatabaseMysqlBase::serverIsReadOnly()

https://gerrit.wikimedia.org/r/583416

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.26; 2020-03-31).Mar 26 2020, 7:00 AM

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2020, 7:10 AM

Anomie closed this task as Resolved.Mar 26 2020, 2:27 PM

Anomie moved this task from Waiting for Review to Done on the Platform Team Workboards (Clinic Duty Team) board.

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM