Page MenuHomePhabricator

AssembleUploadChunksJob triggers: SqlBagOStuff: tries to serialize closure
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error

Request URL: jobrunner.discovery.wmnet /rpc/RunSingleJob.php
Request ID: XTblLApAIDsAALOuNEkAAAEK

message
[{exception_id}] {exception_url}   Exception from line 850 of /srv/mediawiki/php-1.34.0-wmf.14/includes/objectcache/SqlBagOStuff.php: Serialization of 'Closure' is not allowed
trace
 	#0 /srv/mediawiki/php-1.34.0-wmf.14/includes/objectcache/SqlBagOStuff.php(850): serialize(array)
#1 /srv/mediawiki/php-1.34.0-wmf.14/includes/objectcache/SqlBagOStuff.php(444): SqlBagOStuff->serialize(array)
#2 /srv/mediawiki/php-1.34.0-wmf.14/includes/objectcache/SqlBagOStuff.php(388): SqlBagOStuff->updateTableKeys(string, Wikimedia\Rdbms\DatabaseMysqli, string, array, array, string)
#3 /srv/mediawiki/php-1.34.0-wmf.14/includes/objectcache/SqlBagOStuff.php(471): SqlBagOStuff->modifyMulti(array, integer, integer, string)
#4 /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/objectcache/BagOStuff.php(265): SqlBagOStuff->doSet(string, array, integer, integer)
#5 /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/objectcache/ReplicatedBagOStuff.php(84): BagOStuff->set(string, array, integer, integer)
#6 /srv/mediawiki/php-1.34.0-wmf.14/includes/upload/UploadBase.php(2241): ReplicatedBagOStuff->set(string, array, integer)
#7 /srv/mediawiki/php-1.34.0-wmf.14/includes/jobqueue/jobs/AssembleUploadChunksJob.php(103): UploadBase::setSessionStatus(User, string, array)
#8 /srv/mediawiki/php-1.34.0-wmf.14/extensions/EventBus/includes/JobExecutor.php(64): AssembleUploadChunksJob->run()
#9 /srv/mediawiki/rpc/RunSingleJob.php(76): JobExecutor->execute(array)
#10 {main}

Impact

TBD

Notes

This is a new error in mediawiki (not caught be exclusion-filters). I'm hoping it's not serious enough to be a blocker.

Event Timeline

hashar renamed this task from SqlBagOStuff: tries to serialize closure to AassembleUploadChunks triggers: SqlBagOStuff: tries to serialize closure.Jul 23 2019, 1:04 PM

The last change to includes/upload/UploadBase.php has been 4a83841fa742eb0b1ebf5ca068a6ba402c63c0f7 by @aaron . The UploadBase::setSessionStatus call from the stack trace shows up in the diff:

       /**
         * The value will be set in cache for 1 day
         *
+        * Avoid triggering this method on HTTP GET/HEAD requests
+        *
         * @param User $user
         * @param string $statusKey
         * @param array|bool $value
         * @return void
         */
        public static function setSessionStatus( User $user, $statusKey, $value ) {
-               $cache = MediaWikiServices::getInstance()->getMainObjectStash();
-               $key = $cache->makeKey( 'uploadstatus', $user->getId() ?: md5( $user->getName() ), $statusKey );
+               $store = self::getUploadSessionStore();
+               $key = self::getUploadSessionKey( $store, $user, $statusKey );
Krinkle renamed this task from AassembleUploadChunks triggers: SqlBagOStuff: tries to serialize closure to AssembleUploadChunksJob triggers: SqlBagOStuff: tries to serialize closure.Jul 23 2019, 3:34 PM
Krinkle removed a subscriber: Platform Engineering.

The issue is with the array $value being stored, not the method of storage (closure will have been disallowed by the old code as well). Likely triggered by a (recent) commit to a different area in core causing this kind of value to now pass through.

Anomie subscribed.

setSessionStatus() seems like the pathway rather than the cause. I think a more useful course of investigation, if someone can reproduce this on demand, would be to determine where in the array passed to setSessionStatus() (at AssembleUploadChunksJob.php lines 98–104) the Closure is and then figure out where it's coming from.

959daa2ca44c039e72c8a9a5199d4c74dd05caba added the << $status->value = [ 'warnings' => $upload->checkWarnings() ]; >> line. It seems like checkWarnings() has all kinds of File objects inside of it potentially. Some callback could easily slip in given that.

I confirmed that this error is reproducible by doing an async chunked upload of a file with warnings. There are a lot of callers of UploadBase::checkWarnings() in extensions, including some (such as TinyMCE and GWToolset) that depend on the File objects being in there, so that's not an easy thing to change. Maybe it would be better to have a function (say UploadBase::makeWarningsSerializable) which converts the warnings array to something serializable. It would do about half the work that ApiUpload::transformWarnings() is currently doing. Then AssembleUploadChunksJob would call that before storing the array, and ApiUpload::transformWarnings() would be refactored to act on this serializable array. ApiUpload::getApiWarnings(), which calls checkWarnings() separately, would call makeWarningsSerializable() before passing the array to transformWarnings().

Change 525714 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[mediawiki/core@master] Don't try to store File objects to the upload session

https://gerrit.wikimedia.org/r/525714

Change 525714 merged by jenkins-bot:
[mediawiki/core@master] Don't try to store File objects to the upload session

https://gerrit.wikimedia.org/r/525714

Anomie claimed this task.
Anomie reassigned this task from Anomie to tstarling.

Change 526306 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[mediawiki/core@wmf/1.34.0-wmf.15] [1.34.0-wmf.15] Don't try to store File objects to the upload session

https://gerrit.wikimedia.org/r/526306

Change 526306 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.15] [1.34.0-wmf.15] Don't try to store File objects to the upload session

https://gerrit.wikimedia.org/r/526306

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:05 PM