Page MenuHomePhabricator
Create Task
Maniphest T369079

Update `UniqueValueChecker` to query a list of endpoints
Closed, ResolvedPublic
Actions

Description

Problem
With the current work going on around testing the graph split, we investigated which of the constraint checks that are using SPARQL queries are affected.

In an investigation (see T355298 for details) we identified that if we do nothing, UniqueValueChecker will produce some false negatives.

So in order for this to run as required, we need to make UniqueValueChecker query a list of endpoints (and configure that list in production).

Acceptance Criteria

  • UniqueValueChecker runs as normal and produces no false negatives after the graph split

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Replace distinct-values SPARQL queriesmediawiki/extensions/WikibaseQualityConstraintswmf/1.44.0-wmf.19+365 -292
Improve SPARQL query construction in SparqlHelpermediawiki/extensions/WikibaseQualityConstraintswmf/1.44.0-wmf.19+40 -15
Replace distinct-values SPARQL queriesmediawiki/extensions/WikibaseQualityConstraintswmf/1.44.0-wmf.20+365 -292
Improve SPARQL query construction in SparqlHelpermediawiki/extensions/WikibaseQualityConstraintswmf/1.44.0-wmf.20+40 -15
Replace distinct-values SPARQL queriesmediawiki/extensions/WikibaseQualityConstraintsmaster+365 -292
Improve SPARQL query construction in SparqlHelpermediawiki/extensions/WikibaseQualityConstraintsmaster+40 -15
Disable distinct-values constraint checks on Commonsoperations/mediawiki-configmaster+4 -2
Use strict_types in recently modified filesmediawiki/extensions/WikibaseQualityConstraintsmaster+108 -193
UniqueValueChecker queries additional endpointsmediawiki/extensions/WikibaseQualityConstraintsmaster+267 -50
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Lucas_Werkmeister_WMDE updated the task description. (Show Details)Jul 31 2024, 4:21 PM
Lucas_Werkmeister_WMDE subscribed.Jul 31 2024, 4:25 PM
AudreyPenven_WMDE moved this task from In Development to Ready for Peer Review on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.Aug 9 2024, 11:40 AM
ArthurTaylor moved this task from Ready for Peer Review to In Peer Review on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.Aug 13 2024, 9:25 AM
gerritbot added a comment.Aug 13 2024, 4:41 PM

Change #1058598 had a related patch set uploaded (by Audrey Penven; author: Audrey Penven):

[mediawiki/extensions/WikibaseQualityConstraints@master] UniqueValueChecker queries additional endpoints

https://gerrit.wikimedia.org/r/1058598

gerritbot added a project: Patch-For-Review.Aug 13 2024, 4:41 PM
gerritbot added a comment.Aug 15 2024, 11:15 AM

Change #1058598 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@master] UniqueValueChecker queries additional endpoints

https://gerrit.wikimedia.org/r/1058598

Lucas_Werkmeister_WMDE moved this task from In Peer Review to Waiting for Deployment Window on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.Aug 15 2024, 11:21 AM

The code is updated and should hopefully work; once we actually want to use the split graph servers, we’ll still need to update the production config accordingly. (Which might not be 100% trivial, by the way – currently, wgWBQualityConstraintsSparqlEndpoint is set to http://localhost:6009/sparql, not to a public URL like https://query.wikidata.org/sparql. So I’m guessing we’ll need to configure additional envoy proxies?) But at the moment, I don’t think the servers are up and running yet, see T364363.

I’m not sure where this belongs on the board in the meantime. Maybe waiting for deployment window?

Lucas_Werkmeister_WMDE added a comment.Aug 15 2024, 11:22 AM

Oh, or I guess it should stay in peer review (or in development?) while the strict_types patch is still open?

Maintenance_bot removed a project: Patch-For-Review.Aug 15 2024, 11:30 AM
ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.19; 2024-08-20).Aug 15 2024, 12:00 PM
gerritbot added a comment.Aug 15 2024, 3:48 PM

Change #1062965 had a related patch set uploaded (by Audrey Penven; author: Audrey Penven):

[mediawiki/extensions/WikibaseQualityConstraints@master] Use strict_types in recently modified files

https://gerrit.wikimedia.org/r/1062965

gerritbot added a project: Patch-For-Review.Aug 15 2024, 3:48 PM
gerritbot added a comment.Aug 19 2024, 2:34 PM

Change #1062965 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@master] Use strict_types in recently modified files

https://gerrit.wikimedia.org/r/1062965

Lucas_Werkmeister_WMDE closed this task as Resolved.Sep 4 2024, 3:26 PM
Lucas_Werkmeister_WMDE mentioned this in T374021: Make WikibaseQualityConstraints use split-graph query service.
Lucas_Werkmeister_WMDE moved this task from Waiting for Deployment Window to Done on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.

I think it makes the most sense to just close this task and make a separate one for the deployment (without which the task can’t really be verified): T374021: Make WikibaseQualityConstraints use split-graph query service

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2024, 3:31 PM
dcausse reopened this task as Open.Jan 14 2025, 4:51 PM
dcausse subscribed.

Re-opening because the approach taken might not work if the duplicates are spread across the list of endpoints, please see T374021#10458231 for a possible solution.

Lucas_Werkmeister_WMDE claimed this task.Jan 15 2025, 1:30 PM
Lucas_Werkmeister_WMDE moved this task from Done to In Development on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.
Lucas_Werkmeister_WMDE added a subscriber: AudreyPenven_WMDE.
WMDE-leszek subscribed.Jan 15 2025, 3:52 PM
Lucas_Werkmeister_WMDE added a comment.Jan 15 2025, 4:27 PM

Hm, I think one consequence of the current broken query is also that distinct-values constraints are not checked on Commons (or rather, never return a result). Depending on how we implement the code changes, it might start to return Wikidata entities with the same value on Commons constraint check – but, unless we configure an authentication-less WCQS endpoint in the production config, it can never find other Commons (MediaInfo) entities.

My feeling is that such results would almost certainly not be useful. (Finding other Commons entities with the same value might be useful in some cases, though that’s maybe also worth discussing on-wiki.) So for now I think it might be best to completely disable the distinct-values constraint type on Commons, by mapping it to a non-existing item ID (similar to what we do for type constraints – see wgWBQualityConstraintsTypeConstraintId in IS.php). What do you think?

dcausse added a comment.Jan 15 2025, 6:41 PM
In T369079#10463083, @Lucas_Werkmeister_WMDE wrote:

Hm, I think one consequence of the current broken query is also that distinct-values constraints are not checked on Commons (or rather, never return a result). Depending on how we implement the code changes, it might start to return Wikidata entities with the same value on Commons constraint check – but, unless we configure an authentication-less WCQS endpoint in the production config, it can never find other Commons (MediaInfo) entities.

My feeling is that such results would almost certainly not be useful. (Finding other Commons entities with the same value might be useful in some cases, though that’s maybe also worth discussing on-wiki.) So for now I think it might be best to completely disable the distinct-values constraint type on Commons, by mapping it to a non-existing item ID (similar to what we do for type constraints – see wgWBQualityConstraintsTypeConstraintId in IS.php). What do you think?

Good catch, I completely overlooked that constraint checks are running on commons too... I have the impression that we never expected this check to actually work on commons otherwise we'd have created an internal endpoint for it. So I'm all for explicitly disabling it to avoid future confusion.

gerritbot added a comment.Jan 16 2025, 1:58 PM

Change #1112019 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[operations/mediawiki-config@master] Disable distinct-values constraint checks on Commons

https://gerrit.wikimedia.org/r/1112019

gerritbot added a project: Patch-For-Review.Jan 16 2025, 1:58 PM
Tacsipacsi subscribed.Jan 16 2025, 3:04 PM
gerritbot added a comment.Jan 16 2025, 3:07 PM

Change #1112019 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable distinct-values constraint checks on Commons

https://gerrit.wikimedia.org/r/1112019

Stashbot added a comment.Jan 16 2025, 3:07 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T15:07:50Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1112019|Disable distinct-values constraint checks on Commons (T369079)]]

Stashbot added a comment.Jan 16 2025, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T15:13:36Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:1112019|Disable distinct-values constraint checks on Commons (T369079)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Jan 16 2025, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T15:18:42Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1112019|Disable distinct-values constraint checks on Commons (T369079)]] (duration: 10m 51s)

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2025, 3:31 PM
gerritbot added a comment.Jan 17 2025, 11:50 AM

Change #1112198 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseQualityConstraints@master] WIP: Replace distinct-values SPARQL queries

https://gerrit.wikimedia.org/r/1112198

gerritbot added a project: Patch-For-Review.Jan 17 2025, 11:50 AM
gerritbot added a comment.Jan 17 2025, 3:30 PM

Change #1112242 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseQualityConstraints@master] Improve SPARQL query construction in SparqlHelper

https://gerrit.wikimedia.org/r/1112242

Bugreporter mentioned this in T384173: Ignore redirects in unique value check.Jan 20 2025, 7:32 AM
Lucas_Werkmeister_WMDE mentioned this in T384344: Wikibase/Wikidata and WDQS disagree about statement, reference and value namespace prefixes.Jan 21 2025, 5:00 PM
Lucas_Werkmeister_WMDE mentioned this in T269859: Constraint checks assume that query service has latest data.Jan 21 2025, 5:04 PM
Lucas_Werkmeister_WMDE added a comment.Jan 21 2025, 5:06 PM
In T374021#10458231, @Lucas_Werkmeister_WMDE wrote:

Unfortunately it looks like T369079 wasn’t enough, we’ll have to patch the code some more. Currently, the queries in findEntitiesWithSameStatement() and findEntitiesWithSameQualifierOrReference() assume that the subject entity can be found in the graph that we’re querying:

SELECT DISTINCT ?otherEntity WHERE {
  BIND(wds:$guidForRdf AS ?statement)
  BIND(p:$pid AS ?p)
  BIND(ps:$pid AS ?ps)
  ?entity ?p ?statement.
  ?statement ?ps ?value.
  ?otherStatement ?ps ?value.
  ?otherEntity ?p ?otherStatement.
  FILTER(?otherEntity != ?entity)
  MINUS { ?otherStatement wikibase:rank wikibase:DeprecatedRank. }
  $finalSeparatorFilter
}
LIMIT 10

I believe it was originally done this way so that we didn’t have to implement serializing the statement value into the query (though later I added getRdfLiteral() anyway, which we should definitely remove and instead use Wikibase’s RdfBuilder infrastructure). But this doesn’t work for a split query service. We need to refactor the query to not rely on the original entity, and instead directly serialize the statement value into it after all.

It turns out I already wrote much the same thing some four years ago: T269859: Constraint checks assume that query service has latest data

Also, in the above WIP change, I’m trying to construct queries that refer to value nodes by their hash (for the three data value types that get value nodes: globe coordinate, quantity, time), but I’ve now discovered that Wikibase and WDQS disagree about the prefix for those value nodes (v: vs. wdv:), which is going to make it hard to correctly generate those SPARQL queries: T384344: Wikibase/Wikidata and WDQS disagree about statement, reference and value namespace prefixes

Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.Feb 24 2025, 9:20 AM
Lucas_Werkmeister_WMDE moved this task from In Development to Ready for Development on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.
Lucas_Werkmeister_WMDE claimed this task.Mar 6 2025, 11:09 AM
Lucas_Werkmeister_WMDE moved this task from Ready for Development to In Development on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.
Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.Mar 7 2025, 9:19 AM
Lucas_Werkmeister_WMDE moved this task from In Development to Ready for Peer Review on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.
Lucas_Werkmeister_WMDE added a comment.Mar 7 2025, 1:50 PM

To test this locally, I used this code (in the dev tools console) to create a set of properties, one per data type:

P72078 JS code to create properties, one per data type, for testing unique value constraints (T369079)
1mw.loader.using( 'mediawiki.api' ).then( async () => {
2 let api = new mw.Api();
3 for ( const dataType of [
4 'commonsMedia',
5 'entity-schema',
6 'external-id',
7 'wikibase-form',
8 'globe-coordinate',
9 'geo-shape',
10 'wikibase-item',
11 'wikibase-lexeme',
12 'math',
13 'monolingualtext',
14 'musical-notation',
15 'time',
16 'wikibase-property',
17 'quantity',
18 'wikibase-sense',
19 'string',
20 'tabular-data',
21 'url',
22 ] ) {
23 await api.postWithEditToken( {
24 action: 'wbeditentity',
25 new: 'property',
26 data: JSON.stringify( {
27 type: 'property',
28 datatype: dataType,
29 labels: { en: { language: 'en', value: `unique ${dataType} property` } },
30 aliases: { en: [ { language: 'en', value: `${dataType} unique property` } ] },
31 } ),
32 } );
33 }
34} ).catch( console.error );

I then (manually) created two items with the same values for all those properties, loaded a full RDF export of my wiki into a local Blazegraph, configured WBQC to query that blazegraph, and checked constraints on one of those items via Special:ConstraintReport;
image.png (652×1 px, 343 KB)

karapayneWMDE edited projects, added Wikidata-Omega (The Board); removed [Archived]Wikidata Dev Team (Wikidata.org Slice).Mar 7 2025, 4:00 PM
karapayneWMDE moved this task from new to the board to Ready for Peer Review on the Wikidata-Omega (The Board) board.
ArthurTaylor assigned this task to Lucas_Werkmeister_WMDE.Mar 11 2025, 10:10 AM
ArthurTaylor moved this task from Ready for Peer Review to In Development on the Wikidata-Omega (The Board) board.
Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.Mar 11 2025, 10:28 AM
Lucas_Werkmeister_WMDE moved this task from In Development to Ready for Peer Review on the Wikidata-Omega (The Board) board.
gerritbot added a comment.Mar 11 2025, 10:28 AM

Change #1112242 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@master] Improve SPARQL query construction in SparqlHelper

https://gerrit.wikimedia.org/r/1112242

ArthurTaylor claimed this task.Mar 11 2025, 10:43 AM
ArthurTaylor moved this task from Ready for Peer Review to In Peer Review on the Wikidata-Omega (The Board) board.
gerritbot added a comment.Mar 11 2025, 11:01 AM

Change #1112198 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@master] Replace distinct-values SPARQL queries

https://gerrit.wikimedia.org/r/1112198

Lucas_Werkmeister_WMDE removed ArthurTaylor as the assignee of this task.Mar 11 2025, 11:02 AM
Lucas_Werkmeister_WMDE moved this task from In Peer Review to Ready for Tech Verification on the Wikidata-Omega (The Board) board.
Lucas_Werkmeister_WMDE added a subscriber: ArthurTaylor.

We’ll see if this works better than the previous version when we try to make WBQC use the split graph service again.

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2025, 11:31 AM
ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.21; 2025-03-18).Mar 11 2025, 12:00 PM
gerritbot added a comment.Mar 12 2025, 10:55 AM

Change #1126949 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.20] Improve SPARQL query construction in SparqlHelper

https://gerrit.wikimedia.org/r/1126949

gerritbot added a project: Patch-For-Review.Mar 12 2025, 10:55 AM

Change #1126950 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.20] Replace distinct-values SPARQL queries

https://gerrit.wikimedia.org/r/1126950

gerritbot added a comment.Mar 12 2025, 10:55 AM

Change #1126951 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.19] Improve SPARQL query construction in SparqlHelper

https://gerrit.wikimedia.org/r/1126951

gerritbot added a comment.Mar 12 2025, 10:56 AM

Change #1126952 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.19] Replace distinct-values SPARQL queries

https://gerrit.wikimedia.org/r/1126952

gerritbot added a comment.Mar 12 2025, 2:31 PM

Change #1126949 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.20] Improve SPARQL query construction in SparqlHelper

https://gerrit.wikimedia.org/r/1126949

gerritbot added a comment.Mar 12 2025, 2:31 PM

Change #1126950 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.20] Replace distinct-values SPARQL queries

https://gerrit.wikimedia.org/r/1126950

gerritbot added a comment.Mar 12 2025, 2:31 PM

Change #1126951 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.19] Improve SPARQL query construction in SparqlHelper

https://gerrit.wikimedia.org/r/1126951

gerritbot added a comment.Mar 12 2025, 2:32 PM

Change #1126952 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@wmf/1.44.0-wmf.19] Replace distinct-values SPARQL queries

https://gerrit.wikimedia.org/r/1126952

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2025, 2:32 PM
Stashbot added a comment.Mar 12 2025, 2:42 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T14:42:25Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1126949|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126950|Replace distinct-values SPARQL queries (T369079)]], [[gerrit:1126951|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126952|Replace distinct-values SPARQL queries (T369079)]]

Stashbot added a comment.Mar 12 2025, 2:45 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T14:45:30Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:1126949|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126950|Replace distinct-values SPARQL queries (T369079)]], [[gerrit:1126951|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126952|Replace distinct-values SPARQL queries (T369079)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Mar 12 2025, 2:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T14:55:24Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1126949|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126950|Replace distinct-values SPARQL queries (T369079)]], [[gerrit:1126951|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126952|Replace distinct-values SPARQL queries (T369079)]] (duration: 12m 58s)

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.20; 2025-03-11); removed MW-1.44-notes (1.44.0-wmf.21; 2025-03-18).Mar 12 2025, 3:00 PM
karapayneWMDE subscribed.Mar 14 2025, 8:37 AM

task is being tested in https://phabricator.wikimedia.org/T374021

karapayneWMDE closed this task as Resolved.Mar 14 2025, 8:37 AM
karapayneWMDE claimed this task.
karapayneWMDE moved this task from Ready for Tech Verification to In Tech Verification on the Wikidata-Omega (The Board) board.
karapayneWMDE moved this task from In Tech Verification to Done - Moving a task here will move it to the completed tasks workboard, resolve the ticket, and assign it to whomever made the move on the Wikidata-Omega (The Board) board.
karapayneWMDE edited projects, added Wikidata-Omega (Completed Tasks); removed Wikidata-Omega (The Board).
karapayneWMDE moved this task from Newly completed tasks to 2025 Q1 on the Wikidata-Omega (Completed Tasks) board.
karapayneWMDE reassigned this task from karapayneWMDE to LucasWerkmeister.Mar 14 2025, 8:40 AM
karapayneWMDE reassigned this task from LucasWerkmeister to Lucas_Werkmeister_WMDE.
karapayneWMDE added a subscriber: LucasWerkmeister.
LucasWerkmeister unsubscribed.Mar 14 2025, 9:02 AM
Lucas_Werkmeister_WMDE mentioned this in T394456: Expose new split sparql endpoints in api.May 16 2025, 11:47 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits