Page MenuHomePhabricator

RevisionAccessException when trying to import files with FileImporter
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error message
[4263842b-f54e-40ef-ad8b-0165a98b3c10] 2020-07-23 00:00:06: Végzetes kivétel: „MediaWiki\Revision\RevisionAccessException”
Stack Trace
Impact

Unable to import files with FileImporter.

Notes

Reported on-wiki approximately ten minutes after this week’s group1 deployment (~19:25 UTC according to SAL).

The failing request is POSTed, with the following parameters (token changed to XXX):

POST body
automateSourceWikiCleanup=1&intendedRevisionSummary=cleanup&token=XXXC&action=submit&clientUrl=https%3A%2F%2Fen.wikisource.org%2Fwiki%2FFile%3ACentennial_History_of_Oregon_1811-1912%2C_Volume_1.djvu-163.png&intendedFileName=Centennial+History+of+Oregon+1811-1912%2C+Volume+1.djvu-163&intendedWikitext=%3D%3D+%7B%7Bint%3Afiledesc%7D%7D+%3D%3D%0D%0A%7B%7Binformation%0D%0A%7Cdescription%3D%7B%7Ben%7CImage+from+Joseph+Gaston%27s+%27%27Centennial+History+of+Oregon%27%27%7D%7D%0D%0A%7Csource%3D%5Bhttps%3A%2F%2Farchive.org%2Fdownload%2Fcentennialhistor01gast+Centennial+History+of+Oregon%5D%0D%0A%7Cauthor%3D%7B%7Bunknown%7Cauthor%7D%7D%0D%0A%7Cdate%3D%7B%7Bbefore%7C1912%7D%7D%0D%0A%7D%7D%0D%0A%0D%0A%3D%3D+%7B%7Bint%3Alicense-header%7D%7D+%3D%3D%0D%0A%7B%7BPD-US%7D%7D%0D%0A%0D%0A%5B%5BCategory%3ACentennial+History+of+Oregon%5D%5D&actionStats=%7B%22offeredSourceEdit%22%3A1%2C%22editinfo%22%3A1%7D&validationWarnings=%5B%5D&importDetailsHash=509c0e241457d86a9091cb952df0ad3723d2bc52

Event Timeline

Tacsipacsi triaged this task as Unbreak Now! priority.Jul 23 2020, 6:53 AM

This makes the whole FileImporter extension unusable, which is probably big feature loss enough to warrant for UBN.

The error message in the exception says:

Main slot of revision not found in database. See T212428.

I also just convifirmed that the error corresponds. Merging this bug as a duplicate

Also, this is not happening more frequently after the recent releases, although someone should evaluate if the FileImporter function is really broken. So leaving this bug open and at UBN! given I'm not going to go beyond triaging that the original bug is known and already under investigation.

The import fails on German Wikivoyage, too, from both wiki and file.

I'd never seen the issue from T212428 (and I use FileImporter a lot), but in the last 24 hours I've seen this issue reported from enWS, jpWP, and by a Chinese user (not sure which project they call home) on Commons. If it's not actually more prevalent after deploying MediaWiki 1.36/wmf.1 that's a heck of a coincidence!

Yep. The changes in FileImporter this train[1] do contain content model related stuff and if this is popping up now (on non-Wikipedias, as the train hasn't reached them yet) I do believe this was caused by changes this train.

[1] https://www.mediawiki.org/wiki/MediaWiki_1.36/wmf.1/Changelog#FileImporter

Presuming T212428 is an API race condition triggered by replication lag, this seems to be a different issue.

I just tried importing https://en.wikisource.org/wiki/File:Centennial_History_of_Oregon_1811-1912,_Volume_1.djvu-163.png to Commons and got:

[1ba3d98e-192a-4458-9804-e49f64447513] 2020-07-23 15:56:15: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"

The last change to that file was in 2013, and the last change to its information page was 22:38, 22 July 2020. If all replicas are lagged ~20 hours we have a much bigger problem.

I found an instance of this error in Kibana. Some info:

_id: AXN8Y7wk3_NNwgAUMYYj
wiki: commonswiki
reqId: 1ba3d98e-192a-4458-9804-e49f64447513
exception_id: 1ba3d98e-192a-4458-9804-e49f64447513
message: [1ba3d98e-192a-4458-9804-e49f64447513] /wiki/Special:ImportFile MediaWiki\Revision\RevisionAccessException from line 1291 of /srv/mediawiki/php-1.36.0-wmf.1/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
trace: #0 /srv/mediawiki/php-1.36.0-wmf.1/includes/Revision/RevisionStore.php(1281): MediaWiki\Revision\RevisionStore->constructSlotRecords(string, Wikimedia\Rdbms\ResultWrapper, integer, Title, NULL)
#1 /srv/mediawiki/php-1.36.0-wmf.1/includes/Revision/RevisionStore.php(1208): MediaWiki\Revision\RevisionStore->constructSlotRecords(string, Wikimedia\Rdbms\ResultWrapper, integer, Title)
#2 /srv/mediawiki/php-1.36.0-wmf.1/includes/Revision/RevisionStore.php(1330): MediaWiki\Revision\RevisionStore->loadSlotRecords(string, integer, Title)
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /srv/mediawiki/php-1.36.0-wmf.1/includes/Revision/RevisionSlots.php(175): call_user_func(Closure)
#5 /srv/mediawiki/php-1.36.0-wmf.1/includes/Revision/RevisionSlots.php(134): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /srv/mediawiki/php-1.36.0-wmf.1/includes/Revision/RevisionRecord.php(209): MediaWiki\Revision\RevisionSlots->hasSlot(string)
#7 /srv/mediawiki/php-1.36.0-wmf.1/includes/import/ImportableOldRevisionImporter.php(165): MediaWiki\Revision\RevisionRecord->hasSlot(string)
#8 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/Operations/TextRevisionFromTextRevision.php(122): ImportableOldRevisionImporter->import(WikiRevision)
#9 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/Data/ImportOperations.php(133): FileImporter\Operations\TextRevisionFromTextRevision->commit()
#10 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/Data/ImportOperations.php(82): FileImporter\Data\ImportOperations->FileImporter\Data\{closure}(FileImporter\Operations\TextRevisionFromTextRevision)
#11 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/Data/ImportOperations.php(134): FileImporter\Data\ImportOperations->runOperations(integer, integer, boolean, Closure)
#12 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/Services/Importer.php(308): FileImporter\Data\ImportOperations->commit()
#13 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/Services/Importer.php(174): FileImporter\Services\Importer->commitImportOperations(FileImporter\Data\ImportOperations)
#14 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/SpecialImportFile.php(334): FileImporter\Services\Importer->import(User, FileImporter\Data\ImportPlan)
#15 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/SpecialImportFile.php(235): FileImporter\SpecialImportFile->doImport(FileImporter\Data\ImportPlan)
#16 /srv/mediawiki/php-1.36.0-wmf.1/extensions/FileImporter/src/SpecialImportFile.php(200): FileImporter\SpecialImportFile->handleAction(string, FileImporter\Data\ImportPlan)
#17 /srv/mediawiki/php-1.36.0-wmf.1/includes/specialpage/SpecialPage.php(600): FileImporter\SpecialImportFile->execute(NULL)
#18 /srv/mediawiki/php-1.36.0-wmf.1/includes/specialpage/SpecialPageFactory.php(635): SpecialPage->run(NULL)
#19 /srv/mediawiki/php-1.36.0-wmf.1/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#20 /srv/mediawiki/php-1.36.0-wmf.1/includes/MediaWiki.php(940): MediaWiki->performRequest()
#21 /srv/mediawiki/php-1.36.0-wmf.1/includes/MediaWiki.php(543): MediaWiki->main()
#22 /srv/mediawiki/php-1.36.0-wmf.1/index.php(53): MediaWiki->run()
#23 /srv/mediawiki/php-1.36.0-wmf.1/index.php(46): wfIndexMain()
#24 /srv/mediawiki/w/index.php(3): require(string)
#25 {main}

Looking at that, I believe the path I just posted for T212428 should also fix this issue.

The last change to that file was in 2013, and the last change to its information page was 22:38, 22 July 2020. If all replicas are lagged ~20 hours we have a much bigger problem.

The issue is that when importing multiple revisions, the import code will try to load the previous revision immediately after creating it. If replicas are lagged just a little, that will fail, because the revision (or the slot and content data associated with the) are not yet on the replica. A fallback was put in place that would cause the revision to be loaded from the master database in such a case, but that fallback was incomplete. That's what I'm hopefully fixing with my patch.

… the import code will try to load the previous revision immediately after creating it. …

Oh, FileImporter will actually create a new revision, and that's the rev that fails? That makes a whole lot more sense at least.

PS. Should @cicalese have a look at this since their changes in 1.36/wmf.1 seem likely to be what tickled this problem? Or at least something in this week's deploy seems to have suddenly made this much more user-visible.

Change 615798 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] Import: use master DB for loading slots.

https://gerrit.wikimedia.org/r/615798

Change 615753 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] RevisionStore: fall back to master if no revision rows were found

https://gerrit.wikimedia.org/r/615753

PS. Should @cicalese have a look at this since their changes in 1.36/wmf.1 seem likely to be what tickled this problem? Or at least something in this week's deploy seems to have suddenly made this much more user-visible.

@cicalese is on vacation until monday, I worked on this with her in pair-programming. Since this is UBN, we shouldn't wait for her to get back.

I'm pretty sure I found the problem, see my patch.

Oh, FileImporter will actually create a new revision, and that's the rev that fails? That makes a whole lot more sense at least.

It creates a new row in the revision table of the database of the target wiki, representing the old revision.

This should perhaps be in another post so just ignore it if not useful here: I tried to import https://ja.wikipedia.org/wiki/ファイル:コバギボウシ_Hosta_albo-marginata.JPG before FileImporter crashed but the import failed because there is a broken/missing file in filehistory. So I had to upload it "manually". I did not bother to file a bug because there is hopefully not many broken files like this.

Change 615798 merged by jenkins-bot:
[mediawiki/core@master] Import: use master DB for loading slots.

https://gerrit.wikimedia.org/r/615798

Change 616029 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@wmf/1.36.0-wmf.1] Import: use master DB for loading slots.

https://gerrit.wikimedia.org/r/616029

Change 616030 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@REL1_35] Import: use master DB for loading slots.

https://gerrit.wikimedia.org/r/616030

Change 616030 merged by jenkins-bot:
[mediawiki/core@REL1_35] Import: use master DB for loading slots.

https://gerrit.wikimedia.org/r/616030

Fix is merged, the backport to wmf/1.36.0-wmf.1 is ready to be deployed: https://gerrit.wikimedia.org/r/616029

After talking to SRE and RelEng, I'm backporting this change now, let us know if it breaks anything.

Change 616029 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.1] Import: use master DB for loading slots.

https://gerrit.wikimedia.org/r/616029

I just pulled the change on mwdebug1002 and moved https://commons.wikimedia.org/w/index.php?title=File:Centennial_History_of_Oregon_1811-1912,_Volume_1.djvu-163.png and it worked. Deploying the fix to the whole fleet now.

I just pulled the change on mwdebug1002 and moved https://commons.wikimedia.org/w/index.php?title=File:Centennial_History_of_Oregon_1811-1912,_Volume_1.djvu-163.png and it worked. Deploying the fix to the whole fleet now.

Thank you!

Mentioned in SAL (#wikimedia-operations) [2020-07-24T13:00:27Z] <ladsgroup@deploy1001> Synchronized php-1.36.0-wmf.1/includes/import/ImportableOldRevisionImporter.php: [[gerrit:616029|Import: use master DB for loading slots.]] (T258666) (duration: 01m 07s)

It should be fixed now. Try and let me know.

I just moved 50 files from ja.wiki, 10 from vi.wiki and 10 from lv.wiki and no problems! Thanks for fixing! I love you guys/girls! :-)

daniel claimed this task.

Change 615753 merged by jenkins-bot:
[mediawiki/core@master] RevisionStore: fall back to master if no revision rows were found.

https://gerrit.wikimedia.org/r/615753

Change 616608 had a related patch set uploaded (by Cicalese; owner: Daniel Kinzler):
[mediawiki/core@REL1_35] RevisionStore: fall back to master if no revision rows were found.

https://gerrit.wikimedia.org/r/616608

Change 616608 merged by jenkins-bot:
[mediawiki/core@REL1_35] RevisionStore: fall back to master if no revision rows were found.

https://gerrit.wikimedia.org/r/616608