Page MenuHomePhabricator
Create Task
Maniphest T326311

Deletion of Lexemes appears to leak triples related to its forms and senses
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

In T302189 it was reported:

In T302189#8501314, @Nikki wrote:

This report of grammatical features is wrong because it includes deleted data. Like with the previous queries I mentioned, I'm unable to fix it because that takes it from running in under a second to timing out.

This query returns a form which was deleted 11 months ago.

(Here's 100 forms which need cleaning up)

It suggests that forms of deleted lexemes are leaked in the triple store.

Triples whose subjects have a form that is attached to a deleted Lexeme using ontolex:lexicalForm should be deleted as well.
It might be possible that senses are leaked too (attached with ontolex:sense).

AC:

  • when a Lexeme is deleted all its forms and senses should be removed from WDQS.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
deploy version 0.3.156DCaussewikidata/query/deploymaster+16 -18
Fix delete entity queryDCaussewikidata/query/rdfmaster+498 -5
Revert "deploy version 0.3.155"DCaussewikidata/query/deploymaster+18 -16
deploy version 0.3.155DCaussewikidata/query/deploymaster+16 -18
Properly remove forms and senses when deleting lexemesDCaussewikidata/query/rdfmaster+359 -34
Customize query in gerrit

Related Objects

Event Timeline

• dcausse created this task.Jan 5 2023, 1:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2023, 1:33 PM
• dcausse updated the task description. (Show Details)Jan 5 2023, 1:34 PM
Maintenance_bot added a project: Wikidata.Jan 5 2023, 1:45 PM
Mahir256 subscribed.Jan 5 2023, 5:41 PM
Gehel triaged this task as High priority.Jan 9 2023, 4:23 PM
Gehel moved this task from Incoming to Tech Debt on the Wikidata-Query-Service board.
Nikki added a comment.May 25 2024, 10:27 PM

This is still a problem, https://query.wikidata.org/#select%20*%20%7B%20wd:L1144506-F7%20?p%20?v%20%7D returns data that was deleted in August last year - https://www.wikidata.org/wiki/Lexeme:L1144506

Nikki added a comment.Sep 29 2024, 11:25 AM

https://query.wikidata.org/#select%20*%20%7B%20wd:L1071911-S1%20?p%20?v%20%7D has data that was deleted in March last year.

• dcausse moved this task from Tech Debt to Incoming on the Wikidata-Query-Service board.Oct 1 2024, 8:56 AM
dr0ptp4kt moved this task from Incoming to Current work on the Wikidata-Query-Service board.Oct 1 2024, 3:19 PM
dr0ptp4kt edited projects, added: Discovery-Search (Current work); removed: Wikidata-Query-Service.
dr0ptp4kt lowered the priority of this task from High to Medium.Oct 1 2024, 3:26 PM
dr0ptp4kt set the point value for this task to 3.
Gehel subscribed.Oct 14 2024, 3:51 PM

First step is investigating to understand if we are missing events (which happens) or if there is a bug in the updater code.

Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.Oct 14 2024, 3:51 PM
• dcausse claimed this task.Dec 16 2024, 10:00 AM
• dcausse moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
gerritbot added a comment.Dec 16 2024, 6:40 PM

Change #1104714 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Properly remove forms and senses when delete lexemes

https://gerrit.wikimedia.org/r/1104714

gerritbot added a project: Patch-For-Review.Dec 16 2024, 6:40 PM
• dcausse moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Dec 16 2024, 6:40 PM
gerritbot added a comment.Dec 17 2024, 3:19 AM

Change #1104714 merged by jenkins-bot:

[wikidata/query/rdf@master] Properly remove forms and senses when deleting lexemes

https://gerrit.wikimedia.org/r/1104714

Maintenance_bot removed a project: Patch-For-Review.Dec 17 2024, 3:30 AM
• dcausse moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Dec 19 2024, 4:16 PM
gerritbot added a comment.Jan 17 2025, 4:29 PM

Change #1112251 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] deploy version 0.3.152

https://gerrit.wikimedia.org/r/1112251

gerritbot added a project: Patch-For-Review.Jan 17 2025, 4:29 PM
bking subscribed.Feb 10 2025, 4:30 PM
Gehel edited projects, added: Discovery-Search (2025.02.10 - 2025.02.28); removed: Discovery-Search (Current work).Feb 11 2025, 2:17 PM
Gehel moved this task from Incoming to To be Deployed on the Discovery-Search (2025.02.10 - 2025.02.28) board.
Gehel edited projects, added: Discovery-Search (2025.03.01 - 2025.03.21); removed: Discovery-Search (2025.02.10 - 2025.02.28).Feb 28 2025, 1:36 PM
Gehel moved this task from Incoming to To be Deployed on the Discovery-Search (2025.03.01 - 2025.03.21) board.
Gehel edited projects, added: Discovery-Search (2025.03.22 - 2025.04.11); removed: Discovery-Search (2025.03.01 - 2025.03.21).Mar 24 2025, 4:47 PM
Gehel moved this task from Incoming to To be Deployed on the Discovery-Search (2025.03.22 - 2025.04.11) board.
gerritbot added a comment.Apr 1 2025, 4:18 AM

Change #1112251 merged by Ryan Kemper:

[wikidata/query/deploy@master] deploy version 0.3.155

https://gerrit.wikimedia.org/r/1112251

Maintenance_bot removed a project: Patch-For-Review.Apr 1 2025, 4:30 AM
gerritbot added a comment.Apr 1 2025, 8:42 AM

Change #1133071 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] Revert "deploy version 0.3.155"

https://gerrit.wikimedia.org/r/1133071

gerritbot added a project: Patch-For-Review.Apr 1 2025, 8:42 AM

Change #1133071 merged by DCausse:

[wikidata/query/deploy@master] Revert "deploy version 0.3.155"

https://gerrit.wikimedia.org/r/1133071

Stashbot added a comment.Apr 1 2025, 8:46 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-01T08:46:05Z] <dcausse@deploy1003> Started deploy [wdqs/wdqs@354b5ac]: revert T326311, deletion query way too slow

Stashbot added a comment.Apr 1 2025, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-01T08:58:21Z] <dcausse@deploy1003> Finished deploy [wdqs/wdqs@354b5ac]: revert T326311, deletion query way too slow (duration: 12m 15s)

Maintenance_bot removed a project: Patch-For-Review.Apr 1 2025, 9:30 AM
• dcausse mentioned this in T390665: wdqs2016 and 2017 not consuming updates.Apr 1 2025, 12:02 PM
• dcausse moved this task from incoming to in progress on the Wikidata board.
• dcausse moved this task from in progress to incoming on the Wikidata board.Apr 1 2025, 12:51 PM
gerritbot added a comment.Apr 1 2025, 1:03 PM

Change #1133134 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Fix delete entity query

https://gerrit.wikimedia.org/r/1133134

gerritbot added a project: Patch-For-Review.Apr 1 2025, 1:03 PM
• dcausse moved this task from To be Deployed to Needs Review on the Discovery-Search (2025.03.22 - 2025.04.11) board.Apr 1 2025, 1:15 PM
• dcausse added a comment.Apr 2 2025, 8:41 AM

I deployed test snapshot of the updater on wdqs2025 (test machine).
For the following query:

select * {
  VALUES ?f {wd:L327769-F1 wd:L326498-F1 wd:L327288-F1 wd:L324701-F1
             wd:L321884-F1 wd:L320004-F1 wd:L316324-F1 wd:L318083-F1
             wd:L327769-S1 wd:L326498-S1 wd:L327288-S1 wd:L324701-S1
             wd:L321884-S1 wd:L320004-S1 wd:L316324-S1 wd:L318083-S1}
  ?f ?p ?o .
} LIMIT 1000

It does properly return no results as opposed to query.wikidata.org which does not have yet the fix deployed.

gerritbot added a comment.Apr 3 2025, 4:50 PM

Change #1133134 merged by jenkins-bot:

[wikidata/query/rdf@master] Fix delete entity query

https://gerrit.wikimedia.org/r/1133134

Maintenance_bot removed a project: Patch-For-Review.Apr 3 2025, 5:31 PM
EBernhardson moved this task from Needs Review to To be Deployed on the Discovery-Search (2025.03.22 - 2025.04.11) board.Apr 3 2025, 9:01 PM
gerritbot added a comment.Apr 7 2025, 2:34 PM

Change #1134710 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] deploy version 0.3.156

https://gerrit.wikimedia.org/r/1134710

gerritbot added a project: Patch-For-Review.Apr 7 2025, 2:34 PM
Gehel edited projects, added: Discovery-Search (2025.04.11 - 2025.05.02); removed: Discovery-Search (2025.03.22 - 2025.04.11).Apr 11 2025, 1:25 PM
Gehel moved this task from Incoming to To be Deployed on the Discovery-Search (2025.04.11 - 2025.05.02) board.
gerritbot added a comment.Apr 15 2025, 9:41 AM

Change #1134710 merged by DCausse:

[wikidata/query/deploy@master] deploy version 0.3.156

https://gerrit.wikimedia.org/r/1134710

Stashbot added a comment.Apr 15 2025, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-15T09:43:22Z] <dcausse@deploy1003> Started deploy [wdqs/wdqs@fe88851]: version 0.3.156 (T326311)

Stashbot added a comment.Apr 15 2025, 9:57 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-15T09:57:54Z] <dcausse@deploy1003> Finished deploy [wdqs/wdqs@fe88851]: version 0.3.156 (T326311) (duration: 14m 31s)

• dcausse closed this task as Resolved.Apr 15 2025, 10:01 AM
• dcausse moved this task from To be Deployed to Done on the Discovery-Search (2025.04.11 - 2025.05.02) board.
Maintenance_bot removed a project: Patch-For-Review.Apr 15 2025, 10:31 AM
Gehel moved this task from Done to Reported on the Discovery-Search (2025.04.11 - 2025.05.02) board.Apr 17 2025, 7:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits