Page MenuHomePhabricator

Unable to serialize r:46: Id 46 out of range.
Closed, DuplicatePublicPRODUCTION ERROR

Description

https://logstash.wikimedia.org/goto/5d4bdd746036a01e3c2dc9d85558f963

Message:

PHP Notice: Unable to unserialize: [a:8:{s:9:"interface";b:1;s:8:"language";s:2:"en";s:3:"key";s:30:"wikimediashoplink-link-tooltip";s:9:"keysToTry";a:1:{i:0;s:30:"wikimediashoplink-link-tooltip";}s:10:"parameters";a:0:{}s:6:"format";s:5:"parse";s:11:"useDatabase";b:1;s:5:"title";r:46;}]. Id 46 out of range.

On 2018-1115 from roughly 13:05:58 to 13:07:03

It happened on all mediawiki servers, apparently just for enwiki which was running 1.33.0-wmf.3. Seems the memcached value is corrupted???? The event self resolved after a minute, maybe related to the cache entry TTL.

#0 /srv/mediawiki/php-1.33.0-wmf.3/includes/Message.php(302): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 [internal function]: Message->unserialize(string)
#2 [internal function]: Memcached->getMultiByKey(string, array, NULL, integer)
#3 /srv/mediawiki/php-1.33.0-wmf.3/includes/libs/objectcache/MemcachedPeclBagOStuff.php(247): Memcached->getMulti(array)
#4 /srv/mediawiki/php-1.33.0-wmf.3/includes/libs/objectcache/WANObjectCache.php(392): MemcachedPeclBagOStuff->getMulti(array)
#5 /srv/mediawiki/php-1.33.0-wmf.3/includes/libs/objectcache/WANObjectCache.php(337): WANObjectCache->getMulti(array, array, array, array)
#6 /srv/mediawiki/php-1.33.0-wmf.3/includes/libs/objectcache/WANObjectCache.php(1192): WANObjectCache->get(string, NULL, array, NULL)
#7 /srv/mediawiki/php-1.33.0-wmf.3/includes/libs/objectcache/WANObjectCache.php(1150): WANObjectCache->doGetWithSetCallback(string, integer, Closure$Skin::buildSidebar;372, array)
#8 /srv/mediawiki/php-1.33.0-wmf.3/includes/skins/Skin.php(1312): WANObjectCache->getWithSetCallback(string, integer, Closure$Skin::buildSidebar;372, array)
#9 /srv/mediawiki/php-1.33.0-wmf.3/includes/skins/SkinTemplate.php(465): Skin->buildSidebar()
#10 /srv/mediawiki/php-1.33.0-wmf.3/includes/skins/SkinTemplate.php(225): SkinTemplate->prepareQuickTemplate()
#11 /srv/mediawiki/php-1.33.0-wmf.3/includes/OutputPage.php(2714): SkinTemplate->outputPage()
#12 /srv/mediawiki/php-1.33.0-wmf.3/includes/MediaWiki.php(869): OutputPage->output(boolean)
#13 /srv/mediawiki/php-1.33.0-wmf.3/includes/MediaWiki.php(881): Closure$MediaWiki::main()
#14 /srv/mediawiki/php-1.33.0-wmf.3/includes/MediaWiki.php(517): MediaWiki->main()
#15 /srv/mediawiki/php-1.33.0-wmf.3/index.php(42): MediaWiki->run()
#16 /srv/mediawiki/w/index.php(3): include(string)
#17 {main}

Event Timeline

That is an issue with the serialization of a variable reference (that is the r in the serialized format). Example:

<?php
$var = "hello";
$arr = [&$var, &$var];
var_dump($arr);
var_dump(serialize($arr));
array(2) {
  [0]=>
  &string(5) "hello"
  [1]=>
  &string(5) "hello"
}
string(30) "a:2:{i:0;s:5:"hello";i:1;R:2;}"

No idea though what happened though. Closing the task for now blaming cosmic rays.

I think I tracked it down: Zend PHP and HHVM behave differently with respect to the Serializable interface and multiple instances of the same object, as shown by https://3v4l.org/J6g8e.

There are three ways serialization works in PHP:

  1. Basic serialization just dumps/restores all the properties of an object.
  2. Using __sleep() and __wakeup(), you can exclude some properties or do other code modifications.
  3. Using the Serializable interface lets you serialize to/from any arbitrary string. Often this string is created by calling serialize() recursively.

So what happens if you have an array with two objects, each of which contains the same third object?

class Foo {
    public $obj;

    public function __construct( $obj ) {
        $this->obj = $obj;
    }
}

$obj = new stdclass;

$foo1 = new Foo( $obj );
$foo2 = new Foo( $obj );

$s = serialize( array( $foo1, $foo2 ) );

$data = unserialize( $s );

With either #1 or #2, it serializes that reference to $obj as a reference (code 'r') so when things are unserialized $data[0]->obj === $data[1]->obj.

without Serializable
a:2:{i:0;O:3:"Foo":1:{s:3:"obj";O:8:"stdClass":0:{}}i:1;O:3:"Foo":1:{s:3:"obj";r:3;}}

Originally, with method #3, it broke the reference so $data[0]->obj and $data[1]->obj would be clones of each other but would not be ===.

Serializable without recursive state
a:2:{i:0;C:3:"Foo":35:{a:1:{s:3:"obj";O:8:"stdClass":0:{}}}i:1;C:3:"Foo":35:{a:1:{s:3:"obj";O:8:"stdClass":0:{}}}}

It seems that was reported as PHP bug 36424. The issue was eventually fixed in PHP 5.4.0 by making recursive calls to serialize() and unserialize() share state.

Serializable with recursive state
a:2:{i:0;C:3:"Foo":35:{a:1:{s:3:"obj";O:8:"stdClass":0:{}}}i:1;C:3:"Foo":20:{a:1:{s:3:"obj";r:4;}}}

Unfortunately it seems HHVM never picked up that fix (HHVM bug 7253).

So probably what's going on here is that someone is testing PHP 7.2 (with X-Wikimedia-Debug or the like) and triggers the creation of a serialized string like "Serializable with recursive state" above. Then when an HHVM request tries to load that state, it blows up because, without having shared state from an outer call to unserialize(), it has nothing for the "r:4" to be a reference to.

A fix for the immediate problem with the sidebar cache would be to adjust Message::serialize() and Message::unserialize() to avoid including Title objects (instead it could serialize the Title as ->getPrefixedText() and unserialize with ::newFromText() or the like). But if this occurs for other cases, they might not be so easily fixed.

This comment was removed by Anomie.

I filed T210528: PHP/HHVM serialization incompatibility in some situations when using Serializable to explain the issue rather than the symptom, so I'm closing this as a duplicate of that task.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM