I'm begging the maintainers to drop the "tar" layer, here: T298436#8704523 . Any changes might come too late for our project, but at least we'll be able to simplify the code if the day comes.

So tar format is really made for streaming, so I am surprised that this is hard to do in your programming language. Seeking is what is problem in tar, but streaming is really easy. It is really just concatenation of files. So it is similar to any other buffered stream.

In T332045#8704761, @Mitar wrote:

So tar format is really made for streaming, so I am surprised that this is hard to do in your programming language. Seeking is what is problem in tar, but streaming is really easy. It is really just concatenation of files. So it is similar to any other buffered stream.

Interesting point! However, the use cases for tar are streaming from a filesystem to an abstract, linear storage (tape) and then streaming from tape onto the filesystem. It's not designed for streaming the *contents* of any individual file through memory and processing without writing to disk first. Although --to-stdout exists and sort-of undermines my point here :-)

I see that Go's tar library and Java's have elegant support for streaming single files from a tarball, so I tend to agree with what you're saying. I guess the question is whether there's any important property that we gain by using tar for these dumps?

In large dumps there are multiple files inside one archive. So tar serves as a standard way to combine those multiple files into one file, and then compression is made over all of that.

In T332045#8705029, @Mitar wrote:

In large dumps there are multiple files inside one archive. So tar serves as a standard way to combine those multiple files into one file, and then compression is made over all of that.

Oh that's really helpful to know, thank you! And if the names are collated then tar's --to-stdout even does the right thing by default, by contcatenating files. I'm starting to think your understanding of tar might be the correct one :-)

Is this splitting behavior (size and naming) documented somewhere I can find? I don't see a place in the internal filename for splits, "hawiki_0.ndjson"

On a tangent, I couldn't run the https://enterprise.wikimedia.com/docs/snapshot/ API without an account, but is this redirecting to the same tarballs, also containing split files?

Yes, I made a library for processing those dumps in Go.

I think I complained somewhere as well that filenames are not documented, but I do not find where. :-) I think they go like hawiki_0.ndjson, hawiki_1.ndjson and so on.

I am not familiar with Snapshot API, I just use public dumps available.

@Mitar I was able to wire up a streaming decoder, so the file format no longer bothers me :-) . Thanks again for the input!