We need a static baseline to compare the output of rewritten queries and document/triage differences if they emerge.
This index should be static and should be compared against similarly static snapshots.
AC:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | gmodena | T422521 WE2.5.6 Data quality and query validation | |||
| Resolved | gmodena | T423403 test infra: deploy main and scholarly blazegraph instances |
To keep things as simple as possible, I'm spinning up a blazegraph docker container
We don't have a system user on these nodes, so I'll run as root.
docker run --user=0 --entrypoint=/runBlazegraph.sh -d -e HEAP_SIZE="110g" -p 9999:9999 --name wdqs\ -v /srv/wdqs/blazegraph:/wdqs/data:z \ -v /srv/tmp/main-20260209/:/wdqs/munge:z \ wikibase/wdqs:wdqs0.3.156
loadData.sh is a bit underdocumented, but I get input params from https://gerrit.wikimedia.org/g/operations/cookbooks/+/9d48e6b04c9e6c5b4cac812473c6ff5d0ed93f00/cookbooks/sre/wdqs/data-reload.py
sudo docker exec -it wdqs bash /wdqs/loadData.sh -n wdq -d /wdqs/munge -f wikidata_main.%04d.nt.gz
Pulling wikibase/wdqs from dockerhub required setting proxies:
$ cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://webproxy.eqiad.wmnet:8080" Environment="HTTPS_PROXY=http://webproxy.eqiad.wmnet:8080" Environment="NO_PROXY=localhost,127.0.0.1,.wmnet,.wikimedia.org $ sudo systemctl daemon-reload $ sudo systemctl restart docker
The 20260209 has been indexed as is available in Blazegraph:
curl -s -X POST -H "Content-Type: application/sparql-query" --data-binary "SELECT (COUNT(*) AS ?count) WHERE {?s ?p ?o}" http://localhost:9999/bigdata/namespace/wdq/sparql -H "Accept: application/sparql-results+json" | jq
{
"head": {
"vars": [
"count"
]
},
"results": {
"bindings": [
{
"count": {
"datatype": "http://www.w3.org/2001/XMLSchema#integer",
"type": "literal",
"value": "8662009796"
}
}
]
}
}$ curl -s -X POST -H "Content-Type: application/sparql-query" --data-binary "SELECT (COUNT(*) AS ?count) WHERE {?s ?p ?o}" http://localhost:9999/bigdata/namespace/wdq/sparql -H "Accept: application/sparql-results+json" | jq { "head": { "vars": [ "count" ] }, "results": { "bindings": [ { "count": { "datatype": "http://www.w3.org/2001/XMLSchema#integer", "type": "literal", "value": "8794607950" } } ] } }