@gmodena we are trying to set up qlever in wmflabs for scholia. While we could successfully test on single nodes, we now are trying to set up a scalable infrastructure using the openstack user interface for the so called "container infrastructure". Behind the ui there is k8s setup. It would be great if we could coordinate the efforts. For example maybe there are some docker images we could reuse, or is there any insides how to use the UI or is there even a way to reuse the same (or similar) kubernetes helm config so we don't need to use the ui? I added you as a user to the wikiqlever project in wmflabs. So you can see our progress. The difference in the goal is we don't believe splitting the graph is a good idea. So we would be as close as possible to the WDQS instances technically, but with a bigger dataset.

Hey @Physikerwelt,

Thanks for reaching out.

In T414443#11555562, @Physikerwelt wrote:

It would be great if we could coordinate the efforts. For example maybe there are some docker images we could reuse, or is there any insides how to use the UI or is there even a way to reuse the same (or similar) kubernetes helm config so we don't need to use the ui?

We will not target k8s. For the service, we will be targeting bare metal, and we want to experiment with high-memory VMs provisioned internally on Ganeti.

For these test hosts we are deploying manually, but the infrastructure will eventually be managed via Puppet, with the database (likely) vendored as a Debian package. Infra code and docs will be publicly available, but I’m afraid we won’t have much overlap with your cloud deployment here.

Might you be interested, the Query UI fronting the new triple store will be deployed on k8s. Work will happen as part of this task’s parent epic. We plan to re-use the current production helmfile wikidata-query-gui that deploys the wmde/wikidata-query-gui image.

In T414443#11557863, @gmodena wrote:

Hey @Physikerwelt,

Thanks for reaching out.

In T414443#11555562, @Physikerwelt wrote:

It would be great if we could coordinate the efforts. For example maybe there are some docker images we could reuse, or is there any insides how to use the UI or is there even a way to reuse the same (or similar) kubernetes helm config so we don't need to use the ui?

We will not target k8s. For the service, we will be targeting bare metal, and we want to experiment with high-memory VMs provisioned internally on Ganeti.

That would be also an option (if not even simpler). However, how do you handle load balancing (or do you have only one machine).

For these test hosts we are deploying manually, but the infrastructure will eventually be managed via Puppet, with the database (likely) vendored as a Debian package. Infra code and docs will be publicly available, but I’m afraid we won’t have much overlap with your cloud deployment here.

Might you be interested, the Query UI fronting the new triple store will be deployed on k8s. Work will happen as part of this task’s parent epic. We plan to re-use the current production helmfile wikidata-query-gui that deploys the wmde/wikidata-query-gui image.

The ui is already running on toolforge (so this is done now).

In T414443#11558023, @Physikerwelt wrote:

That would be also an option (if not even simpler). However, how do you handle load balancing (or do you have only one machine).

wdqs1028 to wdqs1032 won't be load balanced at this stage. They are dev/experimentation hosts that won't be exposed on the public internet with the current configuration.

In production scenarios, including during the migration, WDQS hosts will be behind Wikimedia's load balancing infra.

gmodena updated https://gitlab.wikimedia.org/repos/wikidata-platform/triplestores/-/merge_requests/3

qlever: pin nixpkgs version to 25.11.

gmodena merged https://gitlab.wikimedia.org/repos/wikidata-platform/triplestores/-/merge_requests/3

qlever: pin nixpkgs version to 25.11.

gmodena updated https://gitlab.wikimedia.org/repos/wikidata-platform/triplestores/-/merge_requests/4

Update documentation and include env lockfiles.

gmodena merged https://gitlab.wikimedia.org/repos/wikidata-platform/triplestores/-/merge_requests/4

Update documentation and include env lockfiles.

I ran a few more ingestion over the past week. Here are some stats. Some results are surprising and counter prior experience.

Here I'm ingesting split graph data (20260209 snapshot) generated by the munging process (import_ttl dag). Some stats:

spark-sql (default)> select scope, wiki, count(*) as triples FROM
                   > discovery.wikibase_rdf_subgraphs 
                   > where snapshot = '20260209'
                   > group by scope, wiki
                   > ;

scope	wiki	triples
wikidata_main	wikidata	8662605271
scholarly_articles	wikidata	8794608326

@lerickson @trueg what I find surprising:

1. The delta in number of triples ingested. I did not fine any import failure in either DB log (I pasted output from QLever). This could be a byproduct of how we prep (munge), but requires further investigation. We know we need to review / rethink the munging step anyway (scoped for q4). We need to learn how ingestion works under the hood (e.g. normalization) for both dbs. We need to integrate the prep and indexing process with Data Quality.
2. Significantly faster ingestion times for Virtuoso compared to its baseline. I was not expecting ingestion to be linear with number of triples, but still. One variable that might have influenced Virtuoso (faster than previous tests) is that we have a smaller number of (larger) splits.
3. QLever was slower than Virtuoso. Here I suspect that parallel ingestion did not kick in because we are loading from multiple files. This is not representative.

Regardless: ingestion times are excellent for both regardless of tunables (both db are using relatively vanilla configs). 4 hours to ingest main is a significant improvement to our current run times.

@RKemper @bking do you have some stats on blazegraph ingestion time for splits?

qlever-main

Ingestion time:

$ gmodena@wdqs1029:/srv/wdqs/qlever/index$ time zcat /srv/tmp/main-20260209/wikidata_main.* | ../result/bin/IndexBuilderMain -m 40G -F nt -f - -i wikidata -s ../conf/wikidata.settings.json 

real    425m2.159s
user    3321m18.732s
sys     371m40.150s

Logs at https://phabricator.wikimedia.org/P89832

Number to triples:

$ curl -X POST http://localhost:7001/sparql?access-token=wdqs1029 -H "Content-Type: application/sparql-query" --data-binary "SELECT (COUNT(*)  AS ?count) WHERE {?s ?p ?o}" | jq

{
  "head": {
    "vars": [
      "count"
    ]
  },
  "results": {
    "bindings": [
      {
        "count": {
          "datatype": "http://www.w3.org/2001/XMLSchema#int",
          "type": "literal",
          "value": "8719510057"
        }
      }
    ]
  }
}

qlever-scholarly

Ingestion time;

$ gmodena@wdqs1030:/srv/wdqs/qlever/index$ time zcat /srv/tmp/scholarly-20260209/scholarly_articles.* | ../result/bin/IndexBuilderMain -m 40G -F nt -f - -i wikidata -s ../conf/wikidata.settings.json 

real    419m1.482s
user    3155m20.081s
sys     346m52.428s

Logs at https://phabricator.wikimedia.org/P89831

Number of triples;

{
  "head": {
    "vars": [
      "count"
    ]
  },
  "results": {
    "bindings": [
      {
        "count": {
          "datatype": "http://www.w3.org/2001/XMLSchema#int",
          "type": "literal",
          "value": "9012266805"
        }
      }
    ]
  }
}

virtuoso-main

Ingestion script:

./result/bin/isql 1111 dba dba VERBOSE=OFF PROMPT=OFF <<EOF
ld_dir ('/srv/tmp/main-20260209/', '*.gz', 'http://wikidata.org');
SELECT ll_file, ll_state, ll_error FROM DB.DBA.load_list;
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
wait_for_children;
checkpoint;
set types off;
SELECT ll_file, ll_state, ll_error FROM DB.DBA.load_list WHERE ll_state <> 2 OR NOT isnull(ll_error) ;
exit;
EOF

Ingestion time:

$ gmodena@wdqs1031:/srv/wdqs/virtuoso$ time ./bin/load.sh  
real    260m35.912s
user    0m0.118s
sys     0m0.169s

isql output: https://phabricator.wikimedia.org/P89833

Numebr of triples:

curl -s -X POST -H "Content-Type: application/sparql-query" --data-binary "SELECT (COUNT(*) AS ?count) WHERE {?s ?p ?o}" http://localhost:8890/sparql   -H "Accept: application/sparql-results+json" 
| jq
{
  "head": {
    "link": [],
    "vars": [
      "count"
    ]
  },
  "results": {
    "distinct": false,
    "ordered": true,
    "bindings": [
      {
        "count": {
          "type": "typed-literal",
          "datatype": "http://www.w3.org/2001/XMLSchema#integer",
          "value": "8662014513"
        }
      }
    ]
  }
}

virtuoso-scholarly

Ingestion script:

$ gmodena@wdqs1032:/srv/wdqs/virtuoso$ cat bin/load.sh 
./result/bin/isql 1111 dba dba VERBOSE=OFF PROMPT=OFF <<EOF
ld_dir ('/srv/tmp/scholarly-20260209//', '*.gz', 'http://wikidata.org');
SELECT ll_file, ll_state, ll_error FROM DB.DBA.load_list;
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
rdf_loader_run() &
wait_for_children;
checkpoint;
set types off;
SELECT ll_file, ll_state, ll_error FROM DB.DBA.load_list WHERE ll_state <> 2 OR NOT isnull(ll_error) ;
exit;
EOF

Ingestion time:

$ gmodena@wdqs1032:/srv/wdqs/virtuoso$ time ./bin/load.sh  
real    252m32.785s
user    0m0.105s
sys     0m0.176s

isql output: https://phabricator.wikimedia.org/P89834

Number of triples:

$ curl -s -X POST   -H "Content-Type: application/sparql-query"   --data-binary "SELECT (COUNT(*) AS ?count) WHERE {?s ?p ?o}"   http://localhost:8890/sparql   -H "Accept: application/sparql-results+json" | jq
{
  "head": {
    "link": [],
    "vars": [
      "count"
    ]
  },
  "results": {
    "distinct": false,
    "ordered": true,
    "bindings": [
      {
        "count": {
          "type": "typed-literal",
          "datatype": "http://www.w3.org/2001/XMLSchema#integer",
          "value": "8794610466"
        }
      }
    ]
  }
}

I updated the doc at https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Migration/Development_Infrastructure#eqiad_test_nodes.

endpoints are reachable from the hosts or via ssh tunnel. They are not reachable from other parts of our infra (stat hosts). @trueg we probably need to fine tune network policies for access from dse-k8s, but we can consult SREs for that.

https://www.wikidata.org/wiki/Wikidata:Scaling_Wikidata/Benchmarking/Virtuoso#Run_the_Virtuoso_server_and_load_the_Wikidata_files_into_the_server already notes the slowdown in ingestion rate with Virtuoso as the process proceeds, and postulates a cause.

The parsing part of the QLever ingestion process can be heavily parallelized if the input file or files are well-behaved. Because of this need for well-behaved input files a special flag must be set. The RDF dump files are well-behaved.

In T414443#11691773, @Pfps wrote:

https://www.wikidata.org/wiki/Wikidata:Scaling_Wikidata/Benchmarking/Virtuoso#Run_the_Virtuoso_server_and_load_the_Wikidata_files_into_the_server already notes the slowdown in ingestion rate with Virtuoso as the process proceeds, and postulates a cause.

The slow down I understand. I don't know the internal of the ingestion process, but your assumption seems consistent with memory allocation patterns we saw in https://phabricator.wikimedia.org/T414559#11537749 . Now, we can't compare because the hardware is different, the wdqs1028 host had flaky drives, the virtuoso config is not properly tuned (especially re-memory, which is key here) for 20B triples, (and, for reference, QLever was reading from a nfs share).

That said, I am surprised by how little time (2.5hours) it took to ingest 8.5B triples. @Pfps did you run any experiment to measure ingestion time as a function of split file size (and thus number of splits) ?

In T414443#11691805, @Pfps wrote:

The parsing part of the QLever ingestion process can be heavily parallelized if the input file or files are well-behaved. Because of this need for well-behaved input files a special flag must be set. The RDF dump files are well-behaved.

https://phabricator.wikimedia.org/P89833

2026-03-09 21:42:33.394 - INFO: You specified "parallel-parsing = true", which enables faster parsing for TTL files with a well-behaved use of newlines
2026-03-09 21:42:33.394 - INFO: You specified "num-triples-per-batch = 10,000,000", choose a lower value if the index builder runs out of memory

I think this might indeed an issue with file format (we indexed nt files), but needs validation, And we need to look tunables (num-triples-per-batch) vs input size. We'll monitor more closely at the next indexing run. The runtime is still good.

@Pfps do you have any thoughts on the results of sanity check (COUNT(*)) reported? My assumption would be that the different counts are a byproduct of inference/reasoning levels, systems triples used for bookeeping, and implementation details of the munging/cleanup step we run on top of the entity dump file (we need to document this better). Does this track?

I was wondering how much we should rely on broad sanity checks like this to identity systemic issues. Would it make sense to compare counts across runs? In your analysis, did you validate if the indexing is idempotent?

Since comparing the internal indexing strategies between vendors isn't feasible, our plan is to focus on verifying query result equivalence (we don't have a task in phabricator for this work yet). We intend to build an automated test (and regression) suite for this, which would become part of our Data Quality tasks to track regressions in the indexing ETLs (this is Wikimedia-wide tooling, not WDQS/SPARQL specific as of right now). The purpose is to compare a db vs Blazegraph, and different versions of the same db against itself (to avoid upgrades breaking the user experience).

With that in mind, I was wondering if you have ideas on the best way to qualitatively design this query equivalence suite.

In T414443#11696818, @gmodena wrote:

In T414443#11691773, @Pfps wrote:

https://www.wikidata.org/wiki/Wikidata:Scaling_Wikidata/Benchmarking/Virtuoso#Run_the_Virtuoso_server_and_load_the_Wikidata_files_into_the_server already notes the slowdown in ingestion rate with Virtuoso as the process proceeds, and postulates a cause.

The slow down I understand. I don't know the internal of the ingestion process, but your assumption seems consistent with memory allocation patterns we saw in https://phabricator.wikimedia.org/T414559#11537749 . Now, we can't compare because the hardware is different, the wdqs1028 host had flaky drives, the virtuoso config is not properly tuned (especially re-memory, which is key here) for 20B triples, (and, for reference, QLever was reading from a nfs share).

That said, I am surprised by how little time (2.5hours) it took to ingest 8.5B triples. @Pfps did you run any experiment to measure ingestion time as a function of split file size (and thus number of splits) ?

I didn't pay close attention to this issue, just that the initial speed of Virtuoso was about four times the speed later on, as measured when loading all of Wikidata as of October 2024. I don't expect that the slowdown is linear and I also expect that the slowdown depends on the working set being larger than available main memory so I can't give much of a guess as to how long Virtuoso would take to load half of Wikidata.

In T414443#11697049, @gmodena wrote:

In T414443#11691805, @Pfps wrote:

The parsing part of the QLever ingestion process can be heavily parallelized if the input file or files are well-behaved. Because of this need for well-behaved input files a special flag must be set. The RDF dump files are well-behaved.

https://phabricator.wikimedia.org/P89833

2026-03-09 21:42:33.394 - INFO: You specified "parallel-parsing = true", which enables faster parsing for TTL files with a well-behaved use of newlines
2026-03-09 21:42:33.394 - INFO: You specified "num-triples-per-batch = 10,000,000", choose a lower value if the index builder runs out of memory

I think this might indeed an issue with file format (we indexed nt files), but needs validation, And we need to look tunables (num-triples-per-batch) vs input size. We'll monitor more closely at the next indexing run. The runtime is still good.

One part of the index process is just reading the files. So there are benefits to having a smaller file and having that compressed. One could spend a lot of time trying to get the best setup. But this isn't something to spend much time on, except to note that less than one day and no errors is almost certainly good enough. (Certainly its much better than for Blazegraph.) I do note that I had failures with Virtuoso due a long-outstanding bug that I think was fixed in 2025.

@Pfps do you have any thoughts on the results of sanity check (COUNT(*)) reported? My assumption would be that the different counts are a byproduct of inference/reasoning levels, systems triples used for bookeeping, and implementation details of the munging/cleanup step we run on top of the entity dump file (we need to document this better). Does this track?

You do have to be careful just counting triples. I'm pretty sure that Virtuoso adds some triples. I don't think that either MillenniumDB or QLever do.

I was wondering how much we should rely on broad sanity checks like this to identity systemic issues. Would it make sense to compare counts across runs? In your analysis, did you validate if the indexing is idempotent?

Idempotency here I guess means having multiple copies of some or all triples in the input stream. I'm pretty sure that QLever has to make sure that these repeated triples don't cause problems because when multiple processes are reading from one file they may end up ingesting a few triples twice. But it would be bad if this kind of idempotency was not the case, as that would violate one of the core tenets of RDF (which is not a core tenet of either Wikidata or Labelled Property Graphs).

Since comparing the internal indexing strategies between vendors isn't feasible, our plan is to focus on verifying query result equivalence (we don't have a task in phabricator for this work yet). We intend to build an automated test (and regression) suite for this, which would become part of our Data Quality tasks to track regressions in the indexing ETLs (this is Wikimedia-wide tooling, not WDQS/SPARQL specific as of right now). The purpose is to compare a db vs Blazegraph, and different versions of the same db against itself (to avoid upgrades breaking the user experience).

With that in mind, I was wondering if you have ideas on the best way to qualitatively design this query equivalence suite.

Comparing results is not trivial because the order of triples in result sets is not determined by the SPARQL 1.1 standard in all cases. And adding ordering would interfere with performance. What I did was very simple - just count the results except if there is only one result and in that case just look at the first field of the single result. That's quick and doesn't need much extra space. But comparing results needs a fixed graph to evaluate the queries on and a way of guessing which result is correct.

Also, I never used munged files with QLever. I don't know whether that would slow down or speed up QLever.

I can check triple counts on my benchmark machine when the current benchmark run finishes, which may take another day or so.