Page MenuHomePhabricator
Create Task
Maniphest T210901

Stale reads for WDQS Updater
Closed, ResolvedPublic
Actions

Description

I suspect that WDQS updater may be reading older RDF data than the latest edit. I.e. I am getting log messages like this:

20:53:35.953 [update 1] WARN  o.wikidata.query.rdf.tool.rdf.Munger - Stale revision on Q5576287: change is 797238340, RDF is 683087222
09:23:21.076 [update 4] WARN  o.wikidata.query.rdf.tool.rdf.Munger - Stale revision on Q132990: change is 797548413, RDF is 791644240
20:14:17.327 [update 8] WARN  o.wikidata.query.rdf.tool.rdf.Munger - Stale revision on Q15277881: change is 802706968, RDF is 736343558

Which means that the Updater knows that last revision was 802706968, and yet when requesting RDF via https://www.wikidata.org/wiki/Special:EntityData/Q15277881.ttl?flavor=dump it got revision ID 736343558.

This is problematic since this means whatever data is updated between 736343558 and 802706968 is lost. It is not result of the caching, since the request has special URL addition that is different for each request (timestamp-driven).

I wonder if there's any way to ensure we get the freshest RDF and not stale information from Wikidata.

Details

SubjectRepoBranchLines +/-
Fix delayed updateswikidata/query/rdfmaster+11 -8
Repeat updates that come in with stale revisionwikidata/query/rdfmaster+180 -29
Customize query in gerrit

Related Objects
Search...

Event Timeline

Smalyshev created this task.Dec 1 2018, 12:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 1 2018, 12:12 AM
Smalyshev triaged this task as High priority.Dec 1 2018, 12:12 AM
Krenair added a subscriber: Krenair.Dec 1 2018, 12:27 AM
Smalyshev updated the task description. (Show Details)Dec 1 2018, 12:29 AM
Addshore added a subscriber: Addshore.Dec 3 2018, 5:41 PM

I wonder if there's any way to ensure we get the freshest RDF and not stale information from Wikidata.

we could add another param to ensure the latest is retrieved?
How often does this happen?
It might be better to only "ensure" the latest is retrieved if it has already failed once? This could possibly be done inside wdqs?
We could also specify a specific rev id to be retrieved as you have that info?
We currently don't enable access to old revisions via any API T40971

Addshore moved this task from incoming to needs discussion or investigation on the Wikidata board.Dec 3 2018, 5:41 PM
Lucas_Werkmeister_WMDE added a subscriber: Lucas_Werkmeister_WMDE.Dec 3 2018, 5:52 PM
In T210901#4794821, @Addshore wrote:

We currently don't enable access to old revisions via any API T40971

I don’t follow – isn’t that what https://www.wikidata.org/wiki/Special:EntityData/Q15277881.ttl?flavor=dump&oldid=802706968 does?

Addshore added a comment.Dec 3 2018, 5:53 PM
In T210901#4794860, @Lucas_Werkmeister_WMDE wrote:
In T210901#4794821, @Addshore wrote:

We currently don't enable access to old revisions via any API T40971

I don’t follow – isn’t that what https://www.wikidata.org/wiki/Special:EntityData/Q15277881.ttl?flavor=dump&oldid=802706968 does?

Oh, so we do already have that :D

Smalyshev added a comment.EditedDec 3 2018, 6:35 PM

we could add another param to ensure the latest is retrieved?

The problem is that if the DB does not have this revision, it can't be retrieved...

We could also specify a specific rev id to be retrieved as you have that info?

I don't want to get the specific one, I want to get the latest one. Retrieving specific one would produce significant performance hit, as it means for 10 quick edits in a row I'd do 10 full updates and not one.

Additionally, replica not (yet) having this revision gets us 404, which is the same as item being deleted. Also not an ideal situation.

Smalyshev mentioned this in T168214: Updater misses updates when two updates happen very close to each other.Dec 4 2018, 6:21 PM
Smalyshev added a parent task: T168214: Updater misses updates when two updates happen very close to each other.
gerritbot added a subscriber: gerritbot.Dec 7 2018, 3:18 AM

Change 478135 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Repeat updates that come in with stale revision

https://gerrit.wikimedia.org/r/478135

gerritbot added a project: Patch-For-Review.Dec 7 2018, 3:18 AM
Smalyshev added a project: User-Smalyshev.Dec 7 2018, 7:29 PM
Smalyshev moved this task from Backlog to In review on the User-Smalyshev board.
Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Dec 10 2018, 7:44 PM
Smalyshev claimed this task.Dec 12 2018, 9:26 AM
gerritbot added a comment.Dec 13 2018, 6:57 PM

Change 478135 merged by jenkins-bot:
[wikidata/query/rdf@master] Repeat updates that come in with stale revision

https://gerrit.wikimedia.org/r/478135

Smalyshev added a project: Discovery-Wikidata-Query-Service-Sprint.Dec 14 2018, 12:56 AM
Smalyshev moved this task from In review to Done on the User-Smalyshev board.
Smalyshev moved this task from Backlog to Done on the Discovery-Wikidata-Query-Service-Sprint board.
gerritbot added a comment.Dec 14 2018, 9:38 PM

Change 479762 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Fix delayed updates

https://gerrit.wikimedia.org/r/479762

gerritbot added a comment.Dec 14 2018, 10:01 PM

Change 479762 merged by jenkins-bot:
[wikidata/query/rdf@master] Fix delayed updates

https://gerrit.wikimedia.org/r/479762

Smalyshev closed this task as Resolved.Dec 19 2018, 7:07 AM

Should be resolved now.

Smalyshev mentioned this in T212550: Implement support for ChronologyProtection in events sent when editing Mediawiki/Wikidata.Dec 22 2018, 1:29 AM
Smalyshev mentioned this in T217897: Reduce / remove the aggessive cache busting behaviour of wdqs-updater.Mar 8 2019, 9:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL