Create a parallel loader to improve load performance for WDQS / Blazegraph

Closed, DeclinedPublic

Actions

Performance measured on dump from 20191202: https://dumps.wikimedia.org/wikidatawiki/entities/20191202/
Baseline tIme to load: 4264m29.914s, 714218864640 bytes

Improvements proposed:

1. One-path loading (when data is loaded into SPO index only and POS, OSP are recreated in parallel afterwards).

One-path time to load: 1755m57.082s (41.2% of baseline), 402815582208 bytes (56.4% of baseline)
Indices recreation: In progress.

2. Data to be loaded is parsed in parallel, creating StatementBuffer instances, which then are queued for load into DB.

To be done.

Changeset is https://gerrit.wikimedia.org/r/#/c/wikidata/query/rdf/+/532373/

@Igorkim78: Thanks! Please see https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines and correct the commit message in Gerrit.

@Aklapper , Thank you! Fixed the commit message.