You can observe this behaviour by running this script: https://github.com/JeroenDeDauw/JsonDumpReader/blob/master/tests/WikidataDumpManualTest.php

Almost certainly related to us using pbzip2 now (instead of just bzip2, which we used initially). I doubt there's anything we can do about this.

If it is really important, we might need to go back to bzip2, possibly with a lower block size because the default already takes 4 hours for a single dump and those aren't getting smaller…

In case supporting PHP for this dump format is not something that will be done, it'd be good to add some note so people don't waste their time on this like I did.

Yeh, I ran into this last night.
Super annoying :/

I looked at this a little bit and the problem here is that both Zend and HHVM use the zlib-like bzip2 functions (http://www.bzip.org/1.0.3/html/zlib-compat.html) which only handle a single bzip2 stream, but out pbzip2 compressed dumps consist of several streams (that's how pbzip2 manages to parallelize).

In order to fix this, both upstreams (Zend and HHVM) would need to change to using the "native" bz2 interface (http://www.bzip.org/1.0.3/html/hl-interface.html). Other hacks might be possible, I didn't look into that though.

Alternatively, the zlib-like functions could be adopted… but I'm not sure how much work that would be or whether that's even desirable.

From Monday on the bzip2 dumps will be written with plain bzip2 again, thus they should be readable again with all sorts of readers.

Shall we close this?