Page MenuHomePhabricator

[WDQS Streaming Updater] Update blazegraph based on the content present in the streaming updater output kafka stream
Closed, ResolvedPublic

Description

AC:

  • a component running close to blazegraph should read the content produced in T251270 and update blazegraph
  • this components should produce metrics:
    • the lag similar to what we monitor with the current updater
    • an approximation of the divergence (actual mutation count vs expect mutation count)
    • some perf/sizing metrics (triples sent/time spent/...)
  • this component should be able to resume its operations (kafka offsets/timestamp support)
  • this component is expected to write to a single blazegraph instance (no H/A)

Event Timeline

dcausse created this task.Apr 28 2020, 1:54 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptApr 28 2020, 1:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
dcausse renamed this task from Add a new updater component to update blazegraph based on the content present in the streaming updater output kafka stream to Update blazegraph based on the content present in the streaming updater output kafka stream.Apr 28 2020, 1:54 PM
dcausse updated the task description. (Show Details)
Zbyszko renamed this task from Update blazegraph based on the content present in the streaming updater output kafka stream to [WDQS Streaming Updater] Update blazegraph based on the content present in the streaming updater output kafka stream.Apr 29 2020, 12:41 PM
dcausse claimed this task.May 5 2020, 12:49 PM
dcausse triaged this task as Medium priority.

Change 596230 had a related patch set uploaded (by DCausse; owner: DCausse):
[wikidata/query/rdf@master] Consume the streaming updater rdf stream and apply it to a rdf store

https://gerrit.wikimedia.org/r/596230

Change 596461 had a related patch set uploaded (by DCausse; owner: DCausse):
[wikidata/query/rdf@master] Add a main class to launch the new updater

https://gerrit.wikimedia.org/r/596461

Change 596484 had a related patch set uploaded (by DCausse; owner: DCausse):
[wikidata/query/rdf@master] Collect various metrics for the stream updater consumer

https://gerrit.wikimedia.org/r/596484

Change 596230 merged by jenkins-bot:
[wikidata/query/rdf@master] Consume the streaming updater rdf stream and apply it to a rdf store

https://gerrit.wikimedia.org/r/596230

Change 596461 merged by jenkins-bot:
[wikidata/query/rdf@master] Add a main class to launch the new updater

https://gerrit.wikimedia.org/r/596461

Change 596484 merged by jenkins-bot:
[wikidata/query/rdf@master] Collect various metrics for the stream updater consumer

https://gerrit.wikimedia.org/r/596484

Gehel closed this task as Resolved.Jul 13 2020, 12:54 PM