Page MenuHomePhabricator

Distribute parallel compressed dumps of Wikidata
Closed, InvalidPublic

Description

Hi,

I am Guillermo Facundo Colunga from the WESO research group at the University of Oviedo. We're working with Scholia to process and gather information about scholarly information from Wikidata. At the time we're processing the information through different techniques but most of them require downloading and uncompress the dumps. We would like to be able to take profit of all the cores we have on our servers to decompress the files and with the current compression format, this is not possible. Would be possible to consider other compression formats for the dumps?

Thanks in advance.

Event Timeline

Lydia_Pintscher subscribed.

We are looking into improving the size of the dumps. This is covered in T88991. I am closing this in favor of the more general ticket.