Exception when trying to convert wikidata dump
Closed, DuplicatePublic
Actions

Assigned To

Authored By

	Smalyshev
	Mar 1 2015, 2:02 AM

Description

Trying to use mwdumper to convert wikidata dump, I get this:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2048
	at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
	at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
	at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:392)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
	at org.mediawiki.importer.XmlDumpReader.readDump(XmlDumpReader.java:88)
	at org.mediawiki.dumper.Dumper.main(Dumper.java:142)

Command line: java -jar ~/mwdumper-1.16.jar --format=sql:1.5 /public/dumps/public/wikidatawiki/20141106/wikidatawiki-20141106-pages-articles.xml.bz2

Related Objects

Duplicates Merged Here: T120665: ERROR 1064: SQL syntax error near ''{{About|the surname GonzÃ¡lez|other uses|Gonzalez (disambiguation)}}\n{{Infobox' at line 1 (MWDumper can't import dumps to current schema)

Event Timeline

Smalyshev created this task.Mar 1 2015, 2:02 AM

Smalyshev raised the priority of this task from to Needs Triage.

Smalyshev updated the task description. (Show Details)

Smalyshev added a project: Utilities-mwdumper.

Smalyshev subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 1 2015, 2:02 AM

Smalyshev set Security to None.Mar 1 2015, 2:04 AM

Smalyshev added subscribers: • brion, daniel.

Aklapper triaged this task as Medium priority.Mar 2 2015, 7:46 AM

@Smalyshev: Is there any "You have an error in your SQL syntax" output too?
All stacktraces look like this, so to fix the scheme support that specific line would be very helpful...

Array out of bounds on utf8 reading sounds familiar, will do a little testing and research.

• brion merged a task: T120665: ERROR 1064: SQL syntax error near ''{{About|the surname GonzÃ¡lez|other uses|Gonzalez (disambiguation)}}\n{{Infobox' at line 1 (MWDumper can't import dumps to current schema).Apr 23 2016, 10:05 AM

• brion added subscribers: gerritbot, Petrb, StudiesWorld.

This appears to be an old bug in the xerces XML parser library, which supposedly is fixed in more recent versions. I'll try updating the local copy.

(Essence of the bug: UTF-8 characters that spanned across buffer boundaries would sometimes cause an error when the bytes to string conversion wanted more bytes than it had available.)

Change 285004 had a related patch set uploaded (by Brion VIBBER):
Update Xerxes to 2.11.0

https://gerrit.wikimedia.org/r/285004

gerritbot added a project: Patch-For-Review.Apr 23 2016, 10:24 AM

That changeset updates Xerces and *should* fix the bug, but I should add a test case before merging.

• brion closed this task as a duplicate of T59236: ERROR 1064: SQL syntax error near ''{{Infobox military person\n|name=Alexander Holle\n|birth_date=27 February 1898\' at line 1 (mwdumper fails to import English WP dump).Apr 23 2016, 4:00 PM

Exception when trying to convert wikidata dumpClosed, DuplicatePublicActions

Description

Related Objects

Event Timeline

Exception when trying to convert wikidata dump
Closed, DuplicatePublic
Actions