Page MenuHomePhabricator
Create Task
Maniphest T201147

Expose constraint violations to WDQS using event queue
Open, MediumPublic13 Estimated Story Points
Actions

Description

As a user I would like to see all constraint violations in WDQS, so I can fix them.

GIVEN a constraint check is executed
WHEN a constraint violation is detected
THEN I should be able to query it with SPARQL.

GIVEN a constraint check is executed
WHEN no constraint violations is detected
THEN I constraint violations should no longer show up when queried with SPARQL

NOTE: At the moment constraints violations are only imported to WDQS if they are cached the moment WDQS pulls the rdfs for constraint violations for an item. There is a race condition between the WDQS poller and the constraints check execution and this is why only a fraction of constraint violations are imported.

When T189458: re-enable wdqs kafka poller is working we can create a dedicated event for when a constraint check is finished, so WDQS poller always pulls cached violations.

Tasks:

  • Create new event in the kafka data pipeline when a constraint check is finished. The event would include the entity ID that was being checked.
  • Listen to event in WDQS and pull the data for that specific item.

Relevant files:
WDQ:
https://github.com/wikimedia/wikidata-query-rdf/blob/728a6cf8665a6535585af3d5805aeda07f3517f6/tools/src/main/java/org/wikidata/query/rdf/tool/change/KafkaPoller.java
Constraints extension:
https://github.com/wikimedia/mediawiki-extensions-WikibaseQualityConstraints

Related Objects
Search...

Event Timeline

• Jonas triaged this task as Medium priority.Aug 3 2018, 9:08 AM
• Jonas created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 3 2018, 9:08 AM
• Jonas removed the point value for this task.Aug 3 2018, 9:12 AM
• Jonas moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.Aug 3 2018, 9:22 AM
MisterSynergy subscribed.Aug 4 2018, 4:45 PM
Salgo60 subscribed.Aug 5 2018, 6:26 AM
• Jonas moved this task from Ready to estimate to Incoming on the Wikidata-Campsite board.Aug 15 2018, 1:10 PM
Addshore updated the task description. (Show Details)Aug 21 2018, 12:17 PM
Addshore updated the task description. (Show Details)Aug 21 2018, 12:44 PM
Addshore added a project: [DEPRECATED] wdwb-tech.Aug 21 2018, 12:52 PM
Addshore removed a subtask: T201150: Regularly run constraint checks for all items.Aug 21 2018, 1:08 PM
Addshore added a parent task: T201150: Regularly run constraint checks for all items.
Addshore edited parent tasks, added: T192565: Find constraint violations; removed: T192567: Expose constraint violations to WDQS.Aug 21 2018, 1:12 PM
WMDE-leszek mentioned this in T189458: re-enable wdqs kafka poller.Aug 21 2018, 1:12 PM
WMDE-leszek updated the task description. (Show Details)Aug 21 2018, 1:15 PM
WMDE-leszek updated the task description. (Show Details)
WMDE-leszek set the point value for this task to 13.Aug 21 2018, 1:17 PM
• Jonas updated the task description. (Show Details)Aug 31 2018, 9:46 AM
Addshore added subscribers: Smalyshev, Addshore.Sep 11 2018, 7:54 AM

@Smalyshev I guess when reloading a query service instance with all of the data you won't really be able to use the api or event buss data to do that?
You'll need a dump?
If so, that should be possible with T204024: Store WikibaseQualityConstraint check data in persistent storage instead of in the cache

Addshore moved this task from Inbox to Investigate & Discuss on the [DEPRECATED] wdwb-tech board.Sep 11 2018, 8:06 AM
Addshore moved this task from Incoming to Needs Work on the Wikidata-Campsite board.Sep 18 2018, 9:32 AM
Addshore moved this task from Investigate & Discuss to Hike Leftovers on the [DEPRECATED] wdwb-tech board.Oct 8 2018, 12:49 PM
Lucas_Werkmeister_WMDE subscribed.Jan 15 2019, 1:40 PM
Addshore mentioned this in T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage.Jan 22 2019, 11:27 AM
Addshore moved this task from Needs Work to Blocked / Waiting / External on the Wikidata-Campsite board.Feb 4 2019, 11:17 AM
Addshore changed the task status from Open to Stalled.Jun 27 2019, 12:24 AM

We could move forward with work in this area now and expose the checks that we do run on edit and cache in a queue for wdqs to pick up.
But it is probably worth waiting for both T204031, T204024 and T214362 to be done, so setting to stalled for now.

Gehel moved this task from Incoming to Feature Requests on the Wikidata-Query-Service board.May 6 2020, 12:46 PM
Addshore moved this task from Hike Leftovers to Blocked on the [DEPRECATED] wdwb-tech board.Jan 22 2021, 10:39 PM
dcausse mentioned this in T274982: Disable fetching constraints from the wdqs updater.Feb 17 2021, 9:41 AM
Addshore removed a project: Wikidata-Campsite.Apr 22 2021, 4:07 PM
Bugreporter added a parent task: T281671: [Epic] Import WikibaseQualityConstraint check data in WDQS.May 3 2021, 2:51 PM
Addshore changed the task status from Stalled to Open.Aug 11 2021, 12:47 PM

This is no longer stalled on having the constraint checks run, as they run after every edit.
However they are still not yet persistently stored.
But un-stalling nonetheless

Sj subscribed.Oct 19 2021, 4:31 PM

Looks promising and important -- usage and quality of constraints will certainly increase once this is resolved.

Addshore unsubscribed.Jun 27 2023, 12:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL