I wonder whether we want to include information about redirects in there or simply leave redirects out? Just leaving them out will be easier to implement and not change the schema (thus not breaking b/c), so I'd suggest going down that road.

(In reply to Marius Hoch from comment #1)

I wonder whether we want to include information about redirects in there or
simply leave redirects out? Just leaving them out will be easier to
implement and not change the schema (thus not breaking b/c), so I'd suggest
going down that road.

An item may have a property value which is a redirected item. Does the JSON dump "resolve" the item value and dump the item ID of the redirect target? If yes, then the JSON dump can leave out redirect information. But if the JSON dump does not resolve a redirect, then there must be a dump for the redirects (maybe in the JSON dump or another file).

See: http://tools.ietf.org/html/rfc6901

(In reply to Ori Livneh from comment #3)

See: http://tools.ietf.org/html/rfc6901

<hoo> thinking about it: That would horribly not work, although it might be semantically nice
<hoo> our JSON has close to 40gb now (or so), thus people usually read it line by line and not as a one json

Maybe we want a simple { id: "Q123", redirect: "Q1" } or something link this? People only care (if at all) for the id of the redirected entity and the target of the redirect.

Ok, we just talked about this and decided to simply leave redirects out of the dump for now.

Having a separate json dump with redirect information might be doable though, if there is desire to have this (please open a new bug then).

Change 173664 had a related patch set uploaded by Hoo man:
Don't include redirects in json dumps

https://gerrit.wikimedia.org/r/173664

Change 173664 merged by jenkins-bot:
Don't include redirects in json dumps

https://gerrit.wikimedia.org/r/173664

On http://dumps.wikimedia.org/other/wikidata, JSON dumps are about 3GB. But today's dump is about half the size at only 763 MB. This issues was to remove redirects, but I doubt that would make the dump half the size. Is all the data in the dump?

In T74678#825032, @Jefft0 wrote:

On http://dumps.wikimedia.org/other/wikidata, JSON dumps are about 3GB. But today's dump is about half the size at only 763 MB. This issues was to remove redirects, but I doubt that would make the dump half the size. Is all the data in the dump?

Something obviously failed, I'm looking into that right now. The incomplete dump has been removed.

I'm afraid there may still be some millions of duplicates in Wikidata json dump.

According to its main page, there are 17.209.354 items in Wikidata. I downloaded a Wikidata entities json dump a few days ago and I expected to find the same number of items there. However, I counted about 20,568,190 lines, 20,565,957 of which are items (e.g. entities with an id starting with "Q"). I suppose this 3 million items excess is made of duplicate items.

Anyway, I haven't been able to find any actual duplicate, but it's hard to search for duplicates in a 69Gb file.

If it matters, the dump I used was latest-all.json.bz2 from https://dumps.wikimedia.org/wikidatawiki/entities/ , with date march 28 2016.

In T74678#2179243, @pere_prlpz wrote:

I'm afraid there may still be some millions of duplicates in Wikidata json dump.

According to its main page, there are 17.209.354 items in Wikidata. I downloaded a Wikidata entities json dump a few days ago and I expected to find the same number of items there. However, I counted about 20,568,190 lines, 20,565,957 of which are items (e.g. entities with an id starting with "Q"). I suppose this 3 million items excess is made of duplicate items.

Anyway, I haven't been able to find any actual duplicate, but it's hard to search for duplicates in a 69Gb file.

If it matters, the dump I used was latest-all.json.bz2 from https://dumps.wikimedia.org/wikidatawiki/entities/ , with date march 28 2016.

That is not a bug, the item count on the main page is only counting items with at least one sitelink or at least one statement (or something along these lines), but the dump contains all items. Also the number on the main page is slightly off all the time (due to caching and they way it is incremented/ decremented).

Thank you. Then the problem is just that Wikidata statistics (or dupm files) should be better documented. It's not just Wikidata main page: neither https://www.wikidata.org/wiki/Special:Statistics gives any clue about how many pages are in Wikidata main namespace.