Page MenuHomePhabricator
Create Task
Maniphest T103670

Several "Duplicate entry for key 'PRIMARY'" errors in enwiki-latest-pages-articles.xml.bz2 (05-Jun-2015 23:45, 11984805689 bytes)
Closed, ResolvedPublic
Actions

Description

The current latest XML English-language Wikipedia dump
enwiki-latest-pages-articles.xml.bz2 (05-Jun-2015 23:45, 11984805689 bytes)
has several duplicated pages.

This leads to errors while trying to read its data and populate a MYSQL database.

For example, I get

Exception in thread "main" java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '614219339' for key 'PRIMARY'
at org.mediawiki.importer.XmlDumpReader.readDump(XmlDumpReader.java:92)

The list of all the IDs that are duplicated is:

  • 614219339
  • 663854862
  • 359952698
  • 301899471
  • 559375953
  • 603392565
  • 544004224
  • 624437388
  • 37733084

I checked some of these duplicated entries and I saw that the XML content from <page> to </page> is identical.

Revisions and Commits

rODUM operations-dumps
rODUM284cce377408 make xml{stubs,abstracts,logs}.py behave as dumpBackup.php does
rODUM4ecc651ffcd2 make xml{stubs,abstracts,logs}.py behave as dumpBackup.php does

Event Timeline

Easy_mungo created this task.Jun 24 2015, 11:07 AM
Easy_mungo raised the priority of this task from to Needs Triage.
Easy_mungo updated the task description. (Show Details)
Easy_mungo added projects: Datasets-Archiving, MediaWiki-libs-Rdbms.
Easy_mungo subscribed.
Restricted Application added subscribers: Hydriz, Aklapper. · View Herald TranscriptJun 24 2015, 11:07 AM
Hydriz edited projects, added Datasets-General-or-Unknown; removed Datasets-Archiving.Jun 24 2015, 1:44 PM
Hydriz set Security to None.
Hydriz added a subscriber: ArielGlenn.
Hydriz unsubscribed.Jun 24 2015, 4:23 PM
ArielGlenn added a comment.Jun 29 2015, 4:45 PM

Can you have a look at the stubs file and see if they are duplicated there?

gnosygnu subscribed.Jul 6 2015, 11:48 PM

As per https://lists.wikimedia.org/pipermail/xmldatadumps-l/2015-July/001149.html, I also observed the same behavior in multiple Swedish wiki dumps, as well as the Spanish Wikipedia dump. I excerpt the relevant portion below.

I checked the stubs for the Swedish wikis and verified that they are duplicated there. See the following links.

Let me know if there's anything else. Thanks.

Example 1:
URL: http://dumps.wikimedia.org/svwikiversity/20150602/svwikiversity-20150602-pages-articles.xml.bz2
Title: Audi m8
ID: 18942
SHA1: gd16v3qkmjr2w2j35zhqitjfg97igjt)
Note: Last article in dump. Repeated twice

Example 2:
URL: http://dumps.wikimedia.org/svwikiquote/20150602/svwikiquote-20150602-pages-articles.xml.bz2
Title: Sommarens tolv månader
ID: 6209
SHA1: 9yibnev7pn3atxicayjoay0ave7pcu6
Note: Last article in dump. Repeated twice

Example 3:
URL: http://dumps.wikimedia.org/svwikibooks/20150602/svwikibooks-20150602-pages-articles.xml.bz2
Title: Topologi/Metriska rum
ID: 10001
SHA1: 5zdkpxflzdxhy7gxclludnlasvl6tw3
Note: Last article in dump. Repeated twice

Example 4:
URL: http://dumps.wikimedia.org/svwikisource/20150602/svwikisource-20150602-pages-articles.xml.bz2
Title: Afhandling om svenska stafsättet/4
ID: 88768
SHA1: 7zyj208ur4vit0t41z7xlftlyl69bo7
Note: Last article in dump. Repeated twice

Example 5:
URL: http://dumps.wikimedia.org/eswiki/20150602/eswiki-20150602-pages-articles.xml.bz2
Title (1): Veguer
Title (2): Promo
Note: duplicates are earlier in the dump (Veguer at the 9% mark and
Promo at the 23% mark). There doesn't seem to be a dupe at the end of
the article.

Unaffected:
* http://dumps.wikimedia.org/svwiki/20150602/svwiki-20150602-pages-articles.xml.bz2
* http://dumps.wikimedia.org/svwiktionary/20150603/svwiktionary-20150603-pages-articles.xml.bz2
* http://dumps.wikimedia.org/svwikinews/20150602/svwikinews-20150602-pages-articles.xml.bz2
jayvdb subscribed.Jul 7 2015, 12:39 AM
wpmirrordev subscribed.Jul 7 2015, 9:55 AM

Many dumps have duplicate final entry:

wikiidsha1
betawikiversity-20150602-stub-meta-current.xml.gz29172qj736ayooxudbk4jwvlufhkrfxztexl
cawikisource-20150602-stub-meta-current.xml.gz381718k8ka9anu9gzrd8hvtvmlplc3en0iim
elwikisource-20150602-stub-meta-current.xml.gz22723c9tyqn20o99k5v2sdutakx1ykqi07lz
foundationwiki-20150602-stub-meta-current.xml.gz101982inhryb63iuh859qv6v44k55ygs8v4zv
glwikisource-20150602-stub-meta-current.xml.gz36629tk9uyd7e73pwbohjg2rrr6f22rf0eh
simplewiktionary-20150602-stub-meta-current.xml.gz37265ekmywczb6olyzg7it8k4twiz17z4pul
zuwiki-20150603-stub-meta-current.xml.gz6679roeo5vuvy4coe53v6aoxqd00i49e9vq
zuwiktionary-20150602-stub-meta-current.xml.gz35440lzqcxomsynenvxr53e17hk7l2jw2xk
wpmirrordev added a comment.Jul 7 2015, 11:46 AM
This comment was removed by wpmirrordev.
ArielGlenn added a comment.Jul 20 2015, 12:18 PM

The duplicate title at the end of a run looks to me like it's gone; that was fixed in https://gerrit.wikimedia.org/r/#/c/216416/
The duplicate in the middle is due to the same page being dumped at the end of one set of stubs and at the beginning of the next set. This is a regression; introduced in https://gerrit.wikimedia.org/r/#/c/215666/ because xmlstubs.py and the other xml*.py scripts dump up to and including the last page or item specified.

ArielGlenn closed this task as Resolved by committing rODUM4ecc651ffcd2: make xml{stubs,abstracts,logs}.py behave as dumpBackup.php does.Aug 3 2015, 7:48 PM
ArielGlenn added a commit: rODUM4ecc651ffcd2: make xml{stubs,abstracts,logs}.py behave as dumpBackup.php does.
ArielGlenn added a commit: rODUM284cce377408: make xml{stubs,abstracts,logs}.py behave as dumpBackup.php does.Mar 30 2016, 8:32 PM
ArielGlenn moved this task from Backlog to Done on the Datasets-General-or-Unknown board.Jul 12 2016, 6:40 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits