Page MenuHomePhabricator
Create Task
Maniphest T292348

MWException from line 974 of WikiImporter.php: Missing text field in import
Open, Needs TriagePublic
Actions

Description

When running import-wikitech.sh manually for T292342: Wikitech and wikitech-static out of sync...

23900 (61.26 pages/sec 61.26 revs/sec)
24000 (60.81 pages/sec 60.81 revs/sec)
Done!
You might want to run rebuildrecentchanges.php to regenerate RecentChanges,
and initSiteStats.php to update page and revision counts
100 (40.26 pages/sec 40.26 revs/sec)
200 (69.97 pages/sec 69.97 revs/sec)
MWException from line 974 of /srv/mediawiki/w/includes/import/WikiImporter.php: Missing text field in import.
#0 /srv/mediawiki/w/includes/import/WikiImporter.php(1134): WikiImporter->makeContent(Object(Title), '830', Array)
#1 /srv/mediawiki/w/includes/import/WikiImporter.php(1111): WikiImporter->processUpload(Array, Array)
#2 /srv/mediawiki/w/includes/import/WikiImporter.php(863): WikiImporter->handleUpload(Array)
#3 /srv/mediawiki/w/includes/import/WikiImporter.php(678): WikiImporter->handlePage()
#4 /srv/mediawiki/w/maintenance/importDump.php(353): WikiImporter->doImport()
#5 /srv/mediawiki/w/maintenance/importDump.php(286): BackupReader->importFromHandle(Resource id #741)
#6 /srv/mediawiki/w/maintenance/importDump.php(130): BackupReader->importFromFile('compress.zlib:/...')
#7 /srv/mediawiki/w/maintenance/doMaintenance.php(112): BackupReader->execute()
#8 /srv/mediawiki/w/maintenance/importDump.php(358): require_once('/srv/mediawiki/...')
#9 {main}
Rebuilding $wgRCMaxAge=7776000 seconds (90 days)

I'm guessing it's from the step of php maintenance/importDump.php --uploads /srv/imports/labswiki-${DATE}.xml.gz

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
WikiImporter: do not fail if upload entry in dump lacks 'text' tagmediawiki/coremaster+2 -0
WikiImporter: do not fail if upload entry in dump lacks 'text' tagmediawiki/coreREL1_39+2 -0
WikiImporter: do not fail if upload entry in dump lacks 'text' tagmediawiki/coreREL1_38+2 -0
WikiImporter: do not fail if upload entry in dump lacks 'text' tagmediawiki/coreREL1_35+2 -0
Customize query in gerrit

Related Objects

Event Timeline

Reedy created this task.Oct 2 2021, 5:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2021, 5:37 PM
Reedy added a project: Dumps-Generation.Oct 2 2021, 5:38 PM
Reedy updated the task description. (Show Details)
ArielGlenn subscribed.Oct 5 2021, 7:57 AM

I don't know who "owns" the wikitech sync setup these days. WMCS nominally, I suppose?

ArielGlenn moved this task from Backlog to Other teams on the Dumps-Generation board.Oct 5 2021, 7:58 AM
Reedy added a comment.Oct 5 2021, 6:00 PM

Yeah.

Not sure at this point whether there's something odd in either the export or the import script (ie a MW bug, which seems possibly more likely), or something wikitech/wikitech-static specific.

Reedy mentioned this in T295266: wikitech-static down.Nov 11 2021, 2:43 AM
Southparkfan subscribed.May 24 2022, 9:18 PM

I hit this bug after applying https://gerrit.wikimedia.org/r/c/mediawiki/core/+/787858 to REL1_35 (which was merged due to T288423). However, the patch provided in T288423#7268918 didn't trigger this bug.

Workaround on REL1_35 (provided you're running REL1_35 with change 787858 merged):

diff --git a/includes/import/WikiImporter.php b/includes/import/WikiImporter.php
index 3b53a1703d..3908082106 100644
--- a/includes/import/WikiImporter.php
+++ b/includes/import/WikiImporter.php
@@ -1070,12 +1070,13 @@ class WikiImporter {
                $revision = new WikiRevision( $this->config );
                $revId = $pageInfo['id'];
                $title = $pageInfo['_title'];
-               $content = $this->makeContent( $title, $revId, $uploadInfo );
+               // $content = $this->makeContent( $title, $revId, $uploadInfo );
+               $text = $uploadInfo['text'] ?? '';

                $revision->setTitle( $title );
                $revision->setID( $revId );
                $revision->setTimestamp( $uploadInfo['timestamp'] );
-               $revision->setContent( SlotRecord::MAIN, $content );
+               $revision->setContent( SlotRecord::MAIN, ContentHandler::makeContent( $text, $title ) );
                $revision->setFilename( $uploadInfo['filename'] );
                if ( isset( $uploadInfo['archivename'] ) ) {
                        $revision->setArchiveName( $uploadInfo['archivename'] );

What is the impact of this workaround, no revision history on file pages?

Southparkfan mentioned this in T288423: maintenance/importDump.php "TypeError from line 95" when --uploads enabled.May 24 2022, 9:24 PM
Southparkfan mentioned this in T304129: Missing text section in upload section from dumpBackup causes importDump to fail.May 24 2022, 9:32 PM
Southparkfan added a comment.EditedJul 10 2022, 12:01 PM

PoC (from dumpBackup.php with --include-files and --uploads, images were imported on this wiki from another wiki): P30997

The exception is thrown in WikiImporter->makeContent() because $uploadInfo does not use contain a 'text' key. In the past, a 'text' key was added if the <upload> element didn't have a 'text' key, but this has not been the case anymore since 787858. $uploadInfo is constructed in WikiImporter->handleUpload() based on the contents of the <upload> element, and since it can't find a 'text' tag inside the <upload> element, it won't add any 'text' key to $uploadInfo either.

WikiExporter->writeUpload() will add almost all tags to the <upload> element that are defined in $normalFields inside WikiImporter->handleUpload(), except for 'text'. Allegedly, the 'text' tag should have existed since introduction of image imports (2008), but even back then it wasn't there. I haven't been able to find a MediaWiki version where the <upload> element contained a 'text' tag. The "set text to '' if it's not found" code was a workaround that was introduced in r83233.

Workaround
Setting the 'text' key to an empty string avoids errors. This is a partial restore of this patch.

diff --git a/includes/import/WikiImporter.php b/includes/import/WikiImporter.php
index d40a39c..ce58c0a 100644
--- a/includes/import/WikiImporter.php
+++ b/includes/import/WikiImporter.php
@@ -1209,6 +1209,7 @@ class WikiImporter {
 		$revision = new WikiRevision( $this->config );
 		$revId = $pageInfo['id'];
 		$title = $pageInfo['_title'];
+		$uploadInfo['text'] = $uploadInfo['text'] ?? '';
 		$content = $this->makeContent( $title, $revId, $uploadInfo );
 
 		$revision->setTitle( $title );
gerritbot added a comment.Jul 10 2022, 12:07 PM

Change 812521 had a related patch set uploaded (by Southparkfan; author: Southparkfan):

[mediawiki/core@master] WikiImporter: do not fail if upload entry in dump misses 'text' tag

https://gerrit.wikimedia.org/r/812521

gerritbot added a project: Patch-For-Review.Jul 10 2022, 12:07 PM
Southparkfan added a comment.Jul 10 2022, 12:14 PM

Somewhat related: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/812522

Southparkfan reopened this task as Open.Jul 10 2022, 12:15 PM
Southparkfan closed this task as a duplicate of T304129: Missing text section in upload section from dumpBackup causes importDump to fail.
Southparkfan merged a task: T304129: Missing text section in upload section from dumpBackup causes importDump to fail.
Southparkfan added subscribers: Celess22, Krinkle.
adrelanos subscribed.Jan 19 2023, 10:26 PM
gerritbot added a comment.Feb 20 2023, 1:07 PM

Change 812521 merged by jenkins-bot:

[mediawiki/core@master] WikiImporter: do not fail if upload entry in dump lacks 'text' tag

https://gerrit.wikimedia.org/r/812521

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2023, 1:10 PM
ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.24; 2023-02-20).Feb 20 2023, 2:00 PM
gerritbot added a comment.Feb 20 2023, 6:00 PM

Change 890451 had a related patch set uploaded (by Reedy; author: Southparkfan):

[mediawiki/core@REL1_39] WikiImporter: do not fail if upload entry in dump lacks 'text' tag

https://gerrit.wikimedia.org/r/890451

gerritbot added a project: Patch-For-Review.Feb 20 2023, 6:00 PM

Change 890452 had a related patch set uploaded (by Reedy; author: Southparkfan):

[mediawiki/core@REL1_38] WikiImporter: do not fail if upload entry in dump lacks 'text' tag

https://gerrit.wikimedia.org/r/890452

gerritbot added a comment.Feb 20 2023, 6:01 PM

Change 890453 had a related patch set uploaded (by Reedy; author: Southparkfan):

[mediawiki/core@REL1_35] WikiImporter: do not fail if upload entry in dump lacks 'text' tag

https://gerrit.wikimedia.org/r/890453

gerritbot added a comment.Feb 20 2023, 6:45 PM

Change 890453 merged by jenkins-bot:

[mediawiki/core@REL1_35] WikiImporter: do not fail if upload entry in dump lacks 'text' tag

https://gerrit.wikimedia.org/r/890453

gerritbot added a comment.Feb 20 2023, 6:47 PM

Change 890452 merged by jenkins-bot:

[mediawiki/core@REL1_38] WikiImporter: do not fail if upload entry in dump lacks 'text' tag

https://gerrit.wikimedia.org/r/890452

gerritbot added a comment.Feb 20 2023, 6:48 PM

Change 890451 merged by jenkins-bot:

[mediawiki/core@REL1_39] WikiImporter: do not fail if upload entry in dump lacks 'text' tag

https://gerrit.wikimedia.org/r/890451

ReleaseTaggerBot added projects: MW-1.39-notes, MW-1.38-notes, MW-1.35-notes.Feb 20 2023, 7:00 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 20 2023, 7:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits