HHVM: could not uncompress value in MemcachedPeclBagOStuff.php on line 124
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	ori
	Jun 4 2014, 12:06 AM

Description

Trace from osmium:

Warning: could not uncompress value in /usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/MemcachedPeclBagOStuff.php on line 124

#0 Memcached->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/MemcachedPeclBagOStuff.php:124]
#1 MemcachedPeclBagOStuff->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:126]
#2 BagOStuff->mergeViaCas(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:111]
#3 BagOStuff->merge(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/jobqueue/JobQueueGroup.php:408]
#4 JobQueueGroup->executeReadyPeriodicTasks(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:76]
#5 RunJobs->execute(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/doMaintenance.php:109]
#6 include(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:281]
#7 include(), called at [/usr/local/apache/common-local/multiversion/MWScript.php:97]
#0 Memcached->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/MemcachedPeclBagOStuff.php:124]\n    #1 MemcachedPeclBagOStuff->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:126]\n    #2 BagOStuff->mergeViaCas(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:111]\n    #3 BagOStuff->merge(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/jobqueue/JobQueueGroup.php:408]\n    #4 JobQueueGroup->executeReadyPeriodicTasks(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:76]\n    #5 RunJobs->execute(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/doMaintenance.php:109]\n    #6 include(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:281]\n    #7 include(), called at [/usr/local/apache/common-local/multiversion/MWScript.php:97]

Version: unspecified
Severity: normal

Details

Reference: bz66104

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:11 AM

• bzimport added a project: MediaWiki-General.

• bzimport set Reference to bz66104.

• bzimport added a subscriber: Unknown Object (MLST).

ori created this task.Jun 4 2014, 12:06 AM

I ran the job loop with tcpdump running in the background to capture memcached gets. Here's a key that triggers the warning: srwiki:jobqueuegroup:taskruns:v1

Minimal testcase:

$memc = new Memcached;
$memc->addServer('localhost', 11211);
$memc->get('srwiki:jobqueuegroup:taskruns:v1');

I saved the raw key via memccat to osmium:/tmp/srwiki_jobqueuegroup_taskruns_v1.raw

Is it using igbinary serialization? I ran out of time.

the flags field is 84, which means MEMC_VAL_COMPRESSED and MEMC_VAL_IS_SERIALIZED

When setting the same value in memcached under HHVM, the resultant key has a flags value of 20 (vs. PHP's 84). I think the figures represent:

MEMC_VAL_COMPRESSED | MEMC_VAL_COMPRESSION_FASTLZ | (MEMC_VAL_IS_SERIALIZED & MEMC_VAL_TYPE_MASK) === 84

MEMC_VAL_COMPRESSED | (MEMC_VAL_IS_SERIALIZED & MEMC_VAL_TYPE_MASK) === 20

So this fingers fastlz compression as the culprit, I think.

HHVM doesn't support fastlz compression. Switching PHP's compression to zlib and re-saving the key still doesn't result in a value HHVM can read. Under HHVM, ZLIB's uncompress() is returning Z_DATA_ERROR, indicating that the input data is corrupted.

I ported "new-style" zlib/fastlz decompression to HHVM, adapting https://github.com/php-memcached-dev/php-memcached/blob/master/php_memcached.c#L3288 into https://github.com/wikimedia/operations-software-hhvm-dev/commit/5b32fd599e8ce76bfe3d13723fd9d66c6e22dccf

Will submit a PR upstream

Submitted: https://github.com/atdt/hhvm/commit/3e97781e6c2e3b52c8647957909ec9308d09dfe6

HHVM: could not uncompress value in MemcachedPeclBagOStuff.php on line 124Closed, ResolvedPublicActions

Description

Details

Event Timeline

HHVM: could not uncompress value in MemcachedPeclBagOStuff.php on line 124
Closed, ResolvedPublic
Actions