Page MenuHomePhabricator
Create Task
Maniphest T220257

dumpBackups.php failing with InvalidArgumentException thrown from RevisionStoreRecord for certain wikis
Closed, ResolvedPublic0 Estimated Story PointsPRODUCTION ERROR
Actions

Description

I've narrowed one case down to the problematic sequence of titles:

dumpsgen@snapshot1009:/mnt/dumpsdata/temp/dumpsgen$ /usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=enwiki --full --stub --report=1 --output=file:/mnt/dumpsdata/temp/dumpsgen/badstubs-history.xml  --skip-header --start=4169926 --skip-footer --end 4169928
2019-04-06 08:09:13: enwiki (ID 256699) 0 pages (0.0|0.0/sec all|curr), 1 revs (25.5|25.5/sec all|curr), ETA 2020-05-15 09:18:22 [max 891190890]
2019-04-06 08:09:13: enwiki (ID 256699) 0 pages (0.0|0.0/sec all|curr), 2 revs (49.6|915.8/sec all|curr), ETA 2019-10-31 11:53:18 [max 891190890]
[13802068f4547aefc7dabdb6] [no req]   InvalidArgumentException from line 100 of /srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStoreRecord.php: The given Title does not belong to page ID 4169927 but actually belongs to 4169926
Backtrace:
#0 /srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStore.php(1820): MediaWiki\Revision\RevisionStoreRecord->__construct(Title, User, CommentStoreComment, stdClass, MediaWiki\Revision\RevisionSlots, boolean)
#1 /srv/mediawiki/php-1.33.0-wmf.23/includes/export/XmlDumpWriter.php(327): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass, integer, Title)
#2 /srv/mediawiki/php-1.33.0-wmf.23/includes/export/WikiExporter.php(485): XmlDumpWriter->writeRevision(stdClass)
#3 /srv/mediawiki/php-1.33.0-wmf.23/includes/export/WikiExporter.php(445): WikiExporter->outputPageStreamBatch(Wikimedia\Rdbms\ResultWrapper, stdClass)
#4 /srv/mediawiki/php-1.33.0-wmf.23/includes/export/WikiExporter.php(269): WikiExporter->dumpPages(string, boolean)
#5 /srv/mediawiki/php-1.33.0-wmf.23/includes/export/WikiExporter.php(154): WikiExporter->dumpFrom(string, boolean)
#6 /srv/mediawiki/php-1.33.0-wmf.23/maintenance/includes/BackupDumper.php(288): WikiExporter->pagesByRange(integer, integer, boolean)
#7 /srv/mediawiki/php-1.33.0-wmf.23/maintenance/dumpBackup.php(81): BackupDumper->dump(integer, integer)
#8 /srv/mediawiki/php-1.33.0-wmf.23/maintenance/doMaintenance.php(94): DumpBackup->execute()
#9 /srv/mediawiki/php-1.33.0-wmf.23/maintenance/dumpBackup.php(138): require_once(string)
#10 /srv/mediawiki/multiversion/MWScript.php(100): require_once(string)
#11 {main}

The underlying issue has always been there but the exception causing stubs on some projects to fail is new.

Details

SubjectRepoBranchLines +/-
for exports, make sure we compare page titles as strings onlymediawiki/corewmf/1.33.0-wmf.24+2 -2
for exports, make sure we compare page titles as strings onlymediawiki/coremaster+2 -2
Customize query in gerrit

Related Objects

Event Timeline

ArielGlenn triaged this task as High priority.Apr 6 2019, 8:47 AM
ArielGlenn created this task.
ArielGlenn added a comment.Apr 6 2019, 9:15 AM

The command in the task description should dump two pages, in order: 2.4 and 2.40 on enwiki.

If either page is dumped by itself, the script completes just fine.

After some scrying around, I tracked it down to this:

WikiExporter.php
        protected function outputPageStreamBatch( $results, $lastRow ) {
                foreach ( $results as $row ) {
                        if ( $lastRow === null ||
                                $lastRow->page_namespace != $row->page_namespace ||
                                $lastRow->page_title != $row->page_title ) {

See that last comparison? Those page titles are both strings, sure, but php does the sloppy compare for us, thinks the rows for both these pages belong to the same page (with title 2.4) and later mw bombs out from that.

Verified that this behavior (treating revisions of '2.40' as belonging to '2.4') existed earlier; all revisions of title '2.40' are included in the page '2.4' in the March 1 2019 stubs dump for enwiki, and no entry is present for page '2.40'; these are old pages and revisions from 2007. In fact in the Nov 2010 copy of the en wiki stubs dump, the same issue is present, never discovered until now.

ArielGlenn updated the task description. (Show Details)Apr 6 2019, 9:16 AM
ArielGlenn added a project: MediaWiki-General.
ArielGlenn added a comment.Apr 6 2019, 9:26 AM

I have live-patched WikiExporter.php on snapshot1009 and am running the dumpBackup command there as it would be run for stubs production (though to a different output file) :

/usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=enwiki --full --stub --report=1000 --output=file:/mnt/dumpsdata/temp/dumpsgen/badstubs-history.xml --output=file:/mnt/dumpsdata/temp/dumpsgen/badstubs-current.xml --filter=latest --output=file:/mnt/dumpsdata/temp/dumpsgen/badstubs-articles.xml --filter=latest --filter=notalk '--filter=namespace:!NS_USER' --skip-header --start=4166862 --skip-footer --end 4186862
gerritbot added a comment.Apr 6 2019, 9:26 AM

Change 501869 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[mediawiki/core@master] for exports, make sure we compare page titles as strings only

https://gerrit.wikimedia.org/r/501869

gerritbot added a project: Patch-For-Review.Apr 6 2019, 9:26 AM
ArielGlenn added a comment.Apr 6 2019, 9:51 AM

The script live-patched ran to completion. I will put this fix out on snapshot1005,6,7 for .wmf23 so that we can get stubs completed. The fix will be overwritten by the first deploy though.

Krinkle added a project: Wikimedia-production-error.Apr 6 2019, 6:34 PM
Krinkle moved this task from Untriaged to Mar 2019 / 1.33wmf20-23 on the Wikimedia-production-error board.
Krinkle edited projects, added MediaWiki-Core-Revision-backend; removed MediaWiki-General.
gerritbot added a comment.Apr 6 2019, 6:53 PM

Change 501869 merged by jenkins-bot:
[mediawiki/core@master] for exports, make sure we compare page titles as strings only

https://gerrit.wikimedia.org/r/501869

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.25; 2019-04-09).Apr 6 2019, 7:01 PM
akosiaris subscribed.Apr 8 2019, 3:51 PM
gerritbot added a comment.Apr 9 2019, 4:36 PM

Change 502537 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[mediawiki/core@wmf/1.33.0-wmf.24] for exports, make sure we compare page titles as strings only

https://gerrit.wikimedia.org/r/502537

gerritbot added a comment.Apr 9 2019, 4:59 PM

Change 502537 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.24] for exports, make sure we compare page titles as strings only

https://gerrit.wikimedia.org/r/502537

ReleaseTaggerBot edited projects, added MW-1.33-notes (1.33.0-wmf.24; 2019-04-02); removed MW-1.33-notes (1.33.0-wmf.25; 2019-04-09).Apr 9 2019, 5:01 PM
ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.Apr 15 2019, 9:54 AM
ArielGlenn mentioned this in T221086: Apr 1 2019 and/or >=1.33-wmf.23 dump run issues.Apr 16 2019, 1:40 PM
Krinkle mentioned this in Blog Post: Production Excellence #9: March 2019.Apr 21 2019, 6:51 PM
ArielGlenn closed this task as Resolved.Apr 22 2019, 7:34 AM
ArielGlenn claimed this task.

While there is probably more that can be done with the dump and export scripts to harden them against exceptions from MW, this specific case has been addressed, fixes are deployed, the error is gone. Closing.

ArielGlenn moved this task from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.May 13 2019, 10:01 AM
ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 28 2019, 11:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits