Page MenuHomePhabricator

stubs dumps (plus all later stages) broken on select wikis
Closed, ResolvedPublic

Description

Reproduce by:

as dumpsgen user from /srv/deployment/dumps/dumps/xmldumps-backup run the following

dumpsgen@snapshot1010:/srv/deployment/dumps/dumps/xmldumps-backup$ /usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=jvwiki --full --stub --report=10 --output=file:/mnt/dumpsdata/temp/dumpsgen/jvwiki-20200520-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/temp/dumpsgen/jvwiki-20200520-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/temp/dumpsgen/jvwiki-20200520-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter='namespace:!NS_USER' --skip-header --start=37324 --skip-footer --end 37326

Produces output:

MediaWiki\Revision\RevisionAccessException from line 1016 of /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/RevisionStore.php: Failed to load data blob from tt:209199: Bad data in text row 209199.
#0 /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/RevisionStore.php(1244): MediaWiki\Revision\RevisionStore->loadSlotContent(Object(MediaWiki\Revision\SlotRecord), NULL, NULL, NULL, 0)
#1 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Object(MediaWiki\Revision\SlotRecord))
#2 /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/SlotRecord.php(307): call_user_func(Object(Closure), Object(MediaWiki\Revision\SlotRecord))
#3 /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\SlotRecord->getContent()
#4 /srv/mediawiki/php-1.35.0-wmf.31/includes/page/WikiPage.php(798): MediaWiki\Revision\RevisionRecord->getContent('main', 1, NULL)
#5 /srv/mediawiki/php-1.35.0-wmf.31/includes/page/WikiPage.php(1056): WikiPage->getContent()
#6 /srv/mediawiki/php-1.35.0-wmf.31/includes/page/WikiPage.php(1043): WikiPage->insertRedirect()
#7 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/XmlDumpWriter.php(244): WikiPage->getRedirectTarget()
#8 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(528): XmlDumpWriter->openPage(Object(stdClass))
#9 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(474): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\ResultWrapper), Object(stdClass))
#10 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(290): WikiExporter->dumpPages('page_id >= 3732...', false)
#11 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(173): WikiExporter->dumpFrom('page_id >= 3732...', false)
#12 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/includes/BackupDumper.php(318): WikiExporter->pagesByRange(37324, 37326, false)
#13 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/dumpBackup.php(82): BackupDumper->dump(1, 1)
#14 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/doMaintenance.php(105): DumpBackup->execute()
#15 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/dumpBackup.php(144): require_once('/srv/mediawiki/...')
#16 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#17 {main}
MediaWiki\Storage\BlobAccessException from line 295 of /srv/mediawiki/php-1.35.0-wmf.31/includes/Storage/SqlBlobStore.php: Bad data in text row 209199.
#0 /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/RevisionStore.php(1014): MediaWiki\Storage\SqlBlobStore->getBlob('tt:209199', 0)
#1 /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/RevisionStore.php(1244): MediaWiki\Revision\RevisionStore->loadSlotContent(Object(MediaWiki\Revision\SlotRecord), NULL, NULL, NULL, 0)
#2 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Object(MediaWiki\Revision\SlotRecord))
#3 /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/SlotRecord.php(307): call_user_func(Object(Closure), Object(MediaWiki\Revision\SlotRecord))
#4 /srv/mediawiki/php-1.35.0-wmf.31/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\SlotRecord->getContent()
#5 /srv/mediawiki/php-1.35.0-wmf.31/includes/page/WikiPage.php(798): MediaWiki\Revision\RevisionRecord->getContent('main', 1, NULL)
#6 /srv/mediawiki/php-1.35.0-wmf.31/includes/page/WikiPage.php(1056): WikiPage->getContent()
#7 /srv/mediawiki/php-1.35.0-wmf.31/includes/page/WikiPage.php(1043): WikiPage->insertRedirect()
#8 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/XmlDumpWriter.php(244): WikiPage->getRedirectTarget()
#9 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(528): XmlDumpWriter->openPage(Object(stdClass))
#10 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(474): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\ResultWrapper), Object(stdClass))
#11 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(290): WikiExporter->dumpPages('page_id >= 3732...', false)
#12 /srv/mediawiki/php-1.35.0-wmf.31/includes/export/WikiExporter.php(173): WikiExporter->dumpFrom('page_id >= 3732...', false)
#13 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/includes/BackupDumper.php(318): WikiExporter->pagesByRange(37324, 37326, false)
#14 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/dumpBackup.php(82): BackupDumper->dump(1, 1)
#15 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/doMaintenance.php(105): DumpBackup->execute()
#16 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/dumpBackup.php(144): require_once('/srv/mediawiki/...')
#17 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#18 {main}

All wikis that have not completed the stubs step and have one or more such revisions will be unable to complete stubs or any following dump steps.

Event Timeline

ArielGlenn triaged this task as Unbreak Now! priority.May 24 2020, 8:19 AM
ArielGlenn created this task.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 24 2020, 8:19 AM

I guess that we need to wrap $page->getRedirectTarget() with invokeLenient() and make sure that no other parts of the code can later trip up on this exception.

Change 598225 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[mediawiki/core@master] Xml dumps should not die when the page redirect target cannot be determined

https://gerrit.wikimedia.org/r/598225

I have tested the above patch on snapshot1010 and the exception is suppressed.

I have manually deployed the above to /srv/mediawiki/php-1.35.0-wmf.31 on all of the snapshot hosts, which should allow stubs to continue for now, but it will break the first time there is a scap.

ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.May 25 2020, 5:13 AM

Change 598225 merged by jenkins-bot:
[mediawiki/core@master] Xml dumps should not die when the page redirect target cannot be determined

https://gerrit.wikimedia.org/r/598225

Change 598564 had a related patch set uploaded (by Tim Starling; owner: ArielGlenn):
[mediawiki/core@wmf/1.35.0-wmf.31] [1.35.0-wmf.31] Xml dumps should not die when the page redirect target cannot be determined

https://gerrit.wikimedia.org/r/598564

Change 598565 had a related patch set uploaded (by Tim Starling; owner: ArielGlenn):
[mediawiki/core@wmf/1.35.0-wmf.32] [wmf/1.35.0-wmf.32] Xml dumps should not die when the page redirect target cannot be determined

https://gerrit.wikimedia.org/r/598565

Change 598564 merged by jenkins-bot:
[mediawiki/core@wmf/1.35.0-wmf.31] [1.35.0-wmf.31] Xml dumps should not die when the page redirect target cannot be determined

https://gerrit.wikimedia.org/r/598564

Change 598565 merged by jenkins-bot:
[mediawiki/core@wmf/1.35.0-wmf.32] [wmf/1.35.0-wmf.32] Xml dumps should not die when the page redirect target cannot be determined

https://gerrit.wikimedia.org/r/598565

Mentioned in SAL (#wikimedia-operations) [2020-05-26T03:53:02Z] <tstarling@deploy1001> Synchronized php-1.35.0-wmf.32/includes/export/XmlDumpWriter.php: T253468 (duration: 01m 07s)

Mentioned in SAL (#wikimedia-operations) [2020-05-26T03:55:21Z] <tstarling@deploy1001> Synchronized php-1.35.0-wmf.31/includes/export/XmlDumpWriter.php: T253468 (duration: 01m 06s)

tstarling closed this task as Resolved.May 26 2020, 3:57 AM
tstarling claimed this task.
tstarling added a subscriber: tstarling.

Deployed, presumed fixed.