Page MenuHomePhabricator

Create dedicated Updater service in Blazegraph
Closed, DeclinedPublic


Right now, Updater sends SPARQL UPDATE queries to Blazegraph in order to update items. Instead, we could create a separate service that would receive:

  1. List of items to update (URIs)
  2. For each item to update, list of triples associated with the item (as collected by Updater)

The service would then perform the same work as Updater did but without the overhead of creating/parsing SPARQL and sending extra data over the wire. For each item, it would:

  1. Fetch the triples that currently belong to the item
    1. Including direct connectors to the item, statement triples, forms/senses (for lexemes) and form statements.
    2. This also collects all reference and value URIs mentioned by any of the statements.
  2. Diff the data above against the data received from Updater.
  3. The triples that is not in incoming set are deleted.
  4. The triples that is in the incoming set but not in result set are inserted
  5. The values/references that had been mentioned in the existing data but not in the incoming data are verified to see if anything links to them. If not, they are deleted.

Having such service would allow us to perform faster updates and thus deliver more scalable service.

Event Timeline

Smalyshev triaged this task as Medium priority.Jan 3 2019, 12:36 AM
Smalyshev created this task.

Change 518760 had a related patch set uploaded (by Smalyshev; owner: Igor Kim):
[wikidata/query/rdf@master] Merging updater (work in progress)

Testing on wdqs-test shows new Updater is 2x faster than old one. Didn't verify validity yet but speed looks good :)

Change 518760 merged by jenkins-bot:
[wikidata/query/rdf@master] Merging updater

Mentioned in SAL (#wikimedia-operations) [2019-11-19T16:15:16Z] <gehel> reloading data from wdqs1007 to wdqs1004 - after failed test of merging updater - T212826

Mentioned in SAL (#wikimedia-operations) [2019-11-19T20:17:46Z] <gehel> completed reloading data from wdqs1007 to wdqs1004 - after failed test of merging updater - T212826

We're moving our efforts to movign the update process to an event streaming style. As such, this dedicated updater isn't really needed anymore.