@Pfps @Physikerwelt this is a placeholder task for further validation work we plan to tackle next quarter. We touched about this in T414443: Setup WDQS instances on test eqiad nodes.. This task is about scaling up bespoke data analysis work we have done so far, automate the tasks, and integrate it with our Data Platform infra. it's purely about qualitative comparison of output, and not about measuring perf.

I would be interested to explore ways in which what we build could be reused outside of Foundation infrastructure. Do you use similar tools in your workflows? Would you find test automation useful? Any pain points you'd like us to be aware of?

I don't think that the problem is solvable in general. You can't even rerun queries and always expect the same results because of updates.

If you are querying the same graph then you can compare result, but even then the comparison is potentially very expensive in general. But for useful Wikidata queries the comparison should be relatively easy, provided that there are no indexicals or randoms, because the Wikidata RDF dump no longer has blank nodes.

What I did was extract a single unit of information from the results and compare that. Incomplete but simple and effective. The details are all in my benchmark code.

In T421298#11752139, @Pfps wrote:

I don't think that the problem is solvable in general. You can't even rerun queries and always expect the same results because of updates.

On test infrastructure we can compare static snapshots (we generate split graph data weekly), updates do not need to be enabled.

That's good. The next determination is whether to do a complete comparison or an incomplete one. Then there is the issue of whether to include a third or fourth engine so that compliance can be estimated.