Page MenuHomePhabricator
Create Task
Maniphest T318329

PHP Warning: Erroneous data format for unserializing 'SiteList'
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   PHP Warning: Erroneous data format for unserializing 'SiteList'
exception.trace
from /srv/mediawiki/php-1.40.0-wmf.2/includes/libs/objectcache/MemcachedPeclBagOStuff.php(196)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.40.0-wmf.2/includes/libs/objectcache/MemcachedPeclBagOStuff.php(196): Memcached->get(string)
#2 /srv/mediawiki/php-1.40.0-wmf.2/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): MemcachedPeclBagOStuff->doGet(string, integer)
#3 /srv/mediawiki/php-1.40.0-wmf.2/includes/libs/objectcache/BagOStuff.php(213): MediumSpecificBagOStuff->get(string, integer)
#4 /srv/mediawiki/php-1.40.0-wmf.2/includes/site/CachingSiteStore.php(111): BagOStuff->getWithSetCallback(string, integer, array)
#5 /srv/mediawiki/php-1.40.0-wmf.2/includes/site/CachingSiteStore.php(128): CachingSiteStore->getSites()
#6 /srv/mediawiki/php-1.40.0-wmf.2/extensions/Wikibase/client/WikibaseClient.ServiceWiring.php(871): CachingSiteStore->getSite(string)
#7 /srv/mediawiki/php-1.40.0-wmf.2/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#8 /srv/mediawiki/php-1.40.0-wmf.2/vendor/wikimedia/services/src/ServiceContainer.php(411): Wikimedia\Services\ServiceContainer->createService(string)
#9 /srv/mediawiki/php-1.40.0-wmf.2/includes/MediaWikiServices.php(301): Wikimedia\Services\ServiceContainer->getService(string)
#10 /srv/mediawiki/php-1.40.0-wmf.2/vendor/wikimedia/services/src/ServiceContainer.php(419): MediaWiki\MediaWikiServices->getService(string)
#11 /srv/mediawiki/php-1.40.0-wmf.2/extensions/Wikibase/client/includes/WikibaseClient.php(241): Wikimedia\Services\ServiceContainer->get(string)
#12 /srv/mediawiki/php-1.40.0-wmf.2/extensions/Wikibase/client/includes/ClientHooks.php(244): Wikibase\Client\WikibaseClient::getSite()
#13 /srv/mediawiki/php-1.40.0-wmf.2/includes/libs/objectcache/BagOStuff.php(216): Wikibase\Client\ClientHooks::Wikibase\Client\{closure}(integer)
#14 /srv/mediawiki/php-1.40.0-wmf.2/extensions/Wikibase/client/includes/ClientHooks.php(258): BagOStuff->getWithSetCallback(string, integer, Closure)
#15 /srv/mediawiki/php-1.40.0-wmf.2/includes/ResourceLoader/FileModule.php(1265): Wikibase\Client\ClientHooks::getLinkitemConfiguration(MediaWiki\ResourceLoader\Context, GlobalVarConfig, NULL)
#16 /srv/mediawiki/php-1.40.0-wmf.2/includes/ResourceLoader/FileModule.php(608): MediaWiki\ResourceLoader\FileModule->expandPackageFiles(MediaWiki\ResourceLoader\Context)
#17 /srv/mediawiki/php-1.40.0-wmf.2/includes/ResourceLoader/Module.php(928): MediaWiki\ResourceLoader\FileModule->getDefinitionSummary(MediaWiki\ResourceLoader\Context)
#18 /srv/mediawiki/php-1.40.0-wmf.2/extensions/WikimediaMaintenance/blameStartupRegistry.php(92): MediaWiki\ResourceLoader\Module->getVersionHash(MediaWiki\ResourceLoader\Context)
#19 /srv/mediawiki/php-1.40.0-wmf.2/maintenance/includes/MaintenanceRunner.php(309): BlameStartupRegistry->execute()
#20 /srv/mediawiki/php-1.40.0-wmf.2/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#21 /srv/mediawiki/php-1.40.0-wmf.2/extensions/WikimediaMaintenance/blameStartupRegistry.php(336): require_once(string)
#22 /srv/mediawiki/multiversion/MWScript.php(120): require_once(string)
#23 {main}
Notes

Continuous, low-frequency stream of errors starting Sept 19th 23:40 UTC. Problem started before the rollout of 1.40.0-wmf.2.

All errors happening on maintenance server mwmaint1002, maximum once per hour around minute 40th, which points at some kind of scheduled process. Seems Wikibase related, all errors originating from Wikidata instance.

Logstash snapshot:
https://logstash.wikimedia.org/goto/4d251594f5b86503047ae53503f42793

Related Objects

Event Timeline

jnuche created this task.Sep 22 2022, 12:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 22 2022, 12:01 PM
Maintenance_bot added a project: Wikidata.Sep 22 2022, 12:29 PM
Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptSep 22 2022, 12:29 PM
WMDE-leszek subscribed.Sep 26 2022, 7:39 AM
Michael subscribed.Sep 26 2022, 8:24 AM
• Lucas_Werkmeister_WMDE subscribed.Sep 26 2022, 10:33 AM

All the errors are from PHP 7.2 according to Logstash – sounds like a PHP 7.4 request serialized sites using the new serialization format (SiteList extends PHP’s ArrayObject, so it would already implement the new serialization format if I’m not mistaken), and PHP 7.2 requests can’t deserialize that? Though it’s strange that this should happen – I thought we patched PHP 7.4 in production to disable writing the new serialization format, and the CachingSiteStore TTL is only one hour (3600 seconds), so by now there shouldn’t be any cached SiteLists left from before we deployed the patched PHP.

• Lucas_Werkmeister_WMDE closed this task as Resolved.Oct 4 2022, 11:13 AM
• Lucas_Werkmeister_WMDE claimed this task.

Hasn’t happened again in a week:

image.png (240×680 px, 14 KB)

This matches up well with T271736#8265154 – maintenance servers were switched to PHP 7.4 a few hours after the last error message in logstash. So I think this was an artifact of the PHP 7.2 → 7.4 migration, and we can consider it resolved.

Krinkle moved this task from Untriaged to Sep 2022 on the Wikimedia-production-error board.Nov 17 2022, 4:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL