"PHP Warning: gzinflate(): data error" on eswiktionary (Nov 2024; 1.44.0-wmf.4)
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Aklapper
	Nov 20 2024, 10:19 AM

Description

See also T296823 and T265989; feel free to close as duplicate.

Error

mwversion: 1.44.0-wmf.4
reqId: 08776f4929182ce61b8640b5
Find reqId in Logstash
CLI call: /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=eswiktionary --dbgroupdefault=dump --full --stub --report=1000 --output=file:/mnt/dumpsdata/xmldatadumps/temp/e/eswiktionary/eswiktionary-20241120-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/xmldatadumps/temp/e/eswiktionary/eswiktionary-20241120-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/xmldatadumps/temp/e/eswiktionary/eswiktionary-20241120-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=1 --skip-footer --end 4319

normalized_message

[{reqId}] {exception_url}   PHP Warning: gzinflate(): data error

Frame	Location	Call
from	/srv/mediawiki/php-1.44.0-wmf.4/includes/Storage/SqlBlobStore.php(652)
#0	[internal function]	MWExceptionHandler::handleError(int, string, string, int, array)
#1	/srv/mediawiki/php-1.44.0-wmf.4/includes/Storage/SqlBlobStore.php(652)	gzinflate(string)
#2	/srv/mediawiki/php-1.44.0-wmf.4/includes/Storage/SqlBlobStore.php(573)	MediaWiki\Storage\SqlBlobStore->decompressData(string, array)
#3	/srv/mediawiki/php-1.44.0-wmf.4/includes/libs/objectcache/WANObjectCache.php(1807)	MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#4	/srv/mediawiki/php-1.44.0-wmf.4/includes/libs/objectcache/WANObjectCache.php(1621)	Wikimedia\ObjectCache\WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#5	/srv/mediawiki/php-1.44.0-wmf.4/includes/Storage/SqlBlobStore.php(575)	Wikimedia\ObjectCache\WANObjectCache->getWithSetCallback(string, int, Closure, array)
#6	/srv/mediawiki/php-1.44.0-wmf.4/includes/Storage/SqlBlobStore.php(348)	MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#7	/srv/mediawiki/php-1.44.0-wmf.4/includes/Storage/SqlBlobStore.php(264)	MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, int)
#8	/srv/mediawiki/php-1.44.0-wmf.4/includes/libs/objectcache/WANObjectCache.php(1807)	MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#9	/srv/mediawiki/php-1.44.0-wmf.4/includes/libs/objectcache/WANObjectCache.php(1621)	Wikimedia\ObjectCache\WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#10	/srv/mediawiki/php-1.44.0-wmf.4/includes/Storage/SqlBlobStore.php(272)	Wikimedia\ObjectCache\WANObjectCache->getWithSetCallback(string, int, Closure, array)
#11	/srv/mediawiki/php-1.44.0-wmf.4/includes/Revision/RevisionStore.php(1210)	MediaWiki\Storage\SqlBlobStore->getBlob(string, int)
#12	/srv/mediawiki/php-1.44.0-wmf.4/includes/Revision/RevisionStore.php(1525)	MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, null, null, null, int)
#13	[internal function]	MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#14	/srv/mediawiki/php-1.44.0-wmf.4/includes/Revision/SlotRecord.php(324)	call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#15	/srv/mediawiki/php-1.44.0-wmf.4/includes/Revision/SlotRecord.php(571)	MediaWiki\Revision\SlotRecord->getContent()
#16	/srv/mediawiki/php-1.44.0-wmf.4/includes/Revision/RevisionSlots.php(205)	MediaWiki\Revision\SlotRecord->getSha1()
#17	[internal function]	MediaWiki\Revision\RevisionSlots::MediaWiki\Revision\{closure}(null, MediaWiki\Revision\SlotRecord)
#18	/srv/mediawiki/php-1.44.0-wmf.4/includes/Revision/RevisionSlots.php(207)	array_reduce(array, Closure, null)
#19	/srv/mediawiki/php-1.44.0-wmf.4/includes/Revision/RevisionStoreRecord.php(167)	MediaWiki\Revision\RevisionSlots->computeSha1()
#20	/srv/mediawiki/php-1.44.0-wmf.4/includes/export/XmlDumpWriter.php(414)	MediaWiki\Revision\RevisionStoreRecord->getSha1()
#21	/srv/mediawiki/php-1.44.0-wmf.4/includes/export/XmlDumpWriter.php(333)	XmlDumpWriter::{closure}()
#22	/srv/mediawiki/php-1.44.0-wmf.4/includes/export/XmlDumpWriter.php(416)	XmlDumpWriter->invokeLenient(Closure, string)
#23	/srv/mediawiki/php-1.44.0-wmf.4/includes/export/WikiExporter.php(541)	XmlDumpWriter->writeRevision(stdClass, array)
#24	/srv/mediawiki/php-1.44.0-wmf.4/includes/export/WikiExporter.php(479)	WikiExporter->outputPageStreamBatch(Wikimedia\Rdbms\MysqliResultWrapper, stdClass)
#25	/srv/mediawiki/php-1.44.0-wmf.4/includes/export/WikiExporter.php(316)	WikiExporter->dumpPages(string, bool)
#26	/srv/mediawiki/php-1.44.0-wmf.4/includes/export/WikiExporter.php(211)	WikiExporter->dumpFrom(string, bool)
#27	/srv/mediawiki/php-1.44.0-wmf.4/maintenance/includes/BackupDumper.php(349)	WikiExporter->pagesByRange(int, int, bool)
#28	/srv/mediawiki/php-1.44.0-wmf.4/maintenance/dumpBackup.php(86)	MediaWiki\Maintenance\BackupDumper->dump(int, int)
#29	/srv/mediawiki/php-1.44.0-wmf.4/maintenance/includes/MaintenanceRunner.php(703)	DumpBackup->execute()
#30	/srv/mediawiki/php-1.44.0-wmf.4/maintenance/run.php(51)	MediaWiki\Maintenance\MaintenanceRunner->run()
#31	/srv/mediawiki/multiversion/MWScript.php(158)	require_once(string)
#32	{main}

Impact

Notes

All 150 occurrences between Nov 20, 2024 @ 10:01:45 and 10:01:53

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Storage: Document error silence reason, use native at	mediawiki/core	master	+17 -19
	Storage: Suppress error from gzinflate and expand log message	mediawiki/core	master	+11 -7

Customize query in gerrit

Related Objects

Mentioned Here: T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production)
T365353: run findBadBlobs.php to mark "known bad" revisions on wmf wikis
T265989: nl.wiktionary.org edits from May 2004 corrupt "PHP Warning: gzinflate(): data error" (fatal RevisionAccessException)
T296823: PHP warnings about bad gzinflate thrown during XML stubs dumps Dec 2021

Event Timeline

Aklapper created this task.Nov 20 2024, 10:19 AM

Restricted Application added a project: Data-Platform. · View Herald TranscriptNov 20 2024, 10:19 AM

When the warnings are new for this wiki that could be happen on the migration on T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production) and maybe losing of gzip information.

In T365353 I have listed bad revisions for that wiki, but no idea if that could be the same or issue.

Maybe @Zabe can have a look if the wiki is already migrated.

eswiktionary finished migrating a bit more than a week ago.

zabe@mwmaint2002:~/text_table_dump$ ls -la | grep eswiktionary
-rwxrwxrwx 1 zabe wikidev  310272064 Nov 11 15:19 eswiktionary
zabe@mwmaint2002:~/text_table_dump$

Sadly logstash isn't really helpful in seeing which revisions are the effected ones. Still I do not really see how it could be related to the MCR migration.

Pppery added a project: Wikimedia-database-issue (Bad data).Nov 22 2024, 7:30 AM

As there was migration on the storage it was an idea that the issue could be from this migration, because ithe error is declared to be new. Have not seen currently changes to the affected classes.

According to the task description this is for the latest revision in the page id range 1 - 4319, all 1491 revisions are accessable by the MediaWiki API, so it seems that everything is correct when reading the revisions via the normal access path.
This could be an issue in the exporter as well. But the exporter does not read the blob addresses, so I would not assume that a blob is loaded and the gzip flag is set, but it is not gzipped (and gzinflate emits the warning).

In the code there is a wfWarn logging "gzinflate() failed", but the SqlBlobStore does not know the revision id at all and not the content address when the decompression happens (that could be changed).

Change #1094534 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Storage: Suppress error from gzinflate and expand log message

https://gerrit.wikimedia.org/r/1094534

gerritbot added a project: Patch-For-Review.Nov 22 2024, 9:44 PM

Change #1094534 merged by jenkins-bot:

[mediawiki/core@master] Storage: Suppress error from gzinflate and expand log message

https://gerrit.wikimedia.org/r/1094534

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.12; 2025-01-14).Dec 18 2024, 3:00 AM

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2024, 3:30 AM

Change #1106012 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] Storage: Document error silence reason, use native at

https://gerrit.wikimedia.org/r/1106012