We could bump memory or we could check whether there is a better API to do the transfer ( See https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/blob/fb94a3c917cb5ac05aa38e7d2e547c2425985cf1/workflow_utils/artifact/cache.py#L127-128 )

Maybe: https://filesystem-spec.readthedocs.io/en/latest/api.html?highlight=clear%20cache#fsspec.utils.read_block ?

Or maybe:

https://filesystem-spec.readthedocs.io/en/stable/features.html#file-buffering-and-random-access

Change the ArtifactSource open method to use a readbuffer cache?

Hm, actually, as far as I can tell, reading from HTTP (and many other sources) uses https://filesystem-spec.readthedocs.io/en/stable/api.html#fsspec.spec.AbstractBufferedFile, which has a default read blocksize of 5MB.

Perhaps this is on the writing side?

No, I guess not:

Buffer only sent on flush() or if buffer is greater than or equal to blocksize.

Oh, or:

https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/http.py#L527-L528

Supports only reading, with read-ahead of a predermined block-size.

In the case that the server does not supply the filesize, only reading of
the complete file in one go is supported.

$ curl -I https://gitlab.wikimedia.org/repos/structured-data/seal/-/package_files/1465/download
...
content-length: 0

So it should work, its just that gitlab package registry doesn't indicate the size of the file?

Ah, good find!

In T348958#9651862, @Ottomata wrote:

$ curl -I https://gitlab.wikimedia.org/repos/structured-data/seal/-/package_files/1465/download
...
content-length: 0

So it should work, its just that gitlab package registry doesn't indicate the size of the file?

Ah good find. That is unfortunate, perhaps we can log it as a gitlab bug?