Background

Looking at the WikidataQualityConstraints (WDQC) code, we can see that it generates Sparql queries via the SparqlHelper class. The constraint checks that trigger these paths are the following:

TypeChecker -> TypeCheckerHelper -> hasClassInRelation / isSubclassOfWithSparqlFallback
FormatChecker -> matchesRegularExpression
UniqueValueChecker -> checkConstraint -> findEntitiesWithSameStatement / findEntitiesWithSameQualifierOrReference

The results are cached, and for some queries a fallback exists that hits the database if the Sparql query fails. For the FormatChecker, the matchesRegularExpression check does not rely on any particular content being in the database / index - this check can be easily tested in all three scenarios. For the TypeChecker and UniqueValueChecker, we require a correspondance between items and properties in the local Wiki and in the index that the QLever server is using.

Local Testing Options

In order to end-to-end test WDQC in our local development environments we need the data in our local wiki to match the data in whatever query service instance we are connecting to. Three possible approaches to local development are outlined below.

Unit testing only

One option to avoid the complexity of a local setup would be to trust that the SparqlHelper is completely abstracting the interface to the QueryService, and just add engine selection and more unit tests to the SparqlHelper. There should only be a limited number of queries involved, and if we develop the QLever support as a beta feature, it would be possible to enable this for testing for specific users in production without breaking anything.

The downside of this approach is that we can only end-to-end test our changes when the train runs, which might introduce multiple 7- to 14-day delays if there are issues with the implementation (though backports might make this a little less bad).

Running QLever locally

The WMF has supplied details of how to run the QLever service (and QS Proxy) locally: https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Migration/Development_Infrastructure

This would allow us to test end-to-end without needing the data on our local system to match the production wikidata.org database. The service setup is moderately complex - developers will need to install nix to build the QLever server from source - and the data would have to be dumped from scratch every time something in the local wiki changes unless we also manage to set up event streams to stream changes from the local wiki to the QLever server (I don’t even know if that is possible without kafka running locally).

The advantage of this approach is that test data we create (per T431789) can be imported to a local server and we can setup the constraint violations locally as we want them, without changing any data in production.

I have added a more complete set of instructions for running a local QLever server for use with WikidataQualityConstraints here: https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Migration/Development_With_Local_Wiki

With the local server running and the items from T431789 created, it should be possible to test QualityConstraints locally, end-to-end.

Import Constraints and properties / items from Wikidata.org and run against the live QLever backend

We can create a fresh local wiki and import items / properties from the server that match the data in production (at wikidata.org). This way, WDQC sparql queries will have the correct item and property IDs to hit the live QLever service.

For this approach to work, we need to find items in production which already have the constraint violations we need - any modifications we make to the data locally will not be reflected by the live QLever QueryService.

The items would need to be exported with Special:Export and imported with Special:Import to ensure that their values are preserved. This means we also need to export/import the constraint items and properties, or edit the imported items to use the local versions of the constraint properties, since the existing maintenance scripts for importing quality constraint items and properties from wikidata.org do not preserve the constraint item and property IDs.

Recommended Approach

Given that we have the plan to work on T431789 and already have the other WikibaseQualityConstraints maintenance scripts for creating constraint properties and items, and detailed instructions for setting the local server up, it seems to me that the easiest approach for end-to-end testing would be "Running QLever locally". That said, I think we would get quite far with just Unit testing, since the number of different types of queries we are generating is quite limited.

I agree with recommended approach, but I'm not sure if we could get quite far with just unit testing. If I'm not mistaken, @AudreyPenven_WMDE found out that some of the tests uses property and item IDs from wikidata.org, therefore could fail on local environment. QLever on local environment will be more than enough anyway