Page MenuHomePhabricator
Create Task
Maniphest T192963

Store Kafka poller position data in the WDQS database
Closed, ResolvedPublic
Actions

Description

Right now, Kafka poller when Updater starts uses timestamp to re-sync with the data stream. While this is the only way to go for dumps, if we're restarting from Kafka polling, we could preserve the current Kafka position and start with it when possible.

Details

SubjectRepoBranchLines +/-
Store Kafka offsets in the DBwikidata/query/rdfmaster+516 -85
Customize query in gerrit

Related Objects
Search...

Event Timeline

Smalyshev created this task.Apr 24 2018, 9:10 PM
Restricted Application added projects: Wikidata, Discovery-ARCHIVED. · View Herald TranscriptApr 24 2018, 9:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev triaged this task as Medium priority.Apr 24 2018, 9:11 PM
gerritbot subscribed.Apr 24 2018, 9:11 PM

Change 421348 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Store Kafka offsets in the DB

https://gerrit.wikimedia.org/r/421348

gerritbot added a project: Patch-For-Review.Apr 24 2018, 9:11 PM
Smalyshev mentioned this in T192871: Enable the query service updater to optionally use RecentChangesLinked for RC input.Apr 24 2018, 9:12 PM
Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Apr 26 2018, 5:44 PM
Smalyshev claimed this task.Apr 28 2018, 7:41 AM
Smalyshev added a parent task: T189458: re-enable wdqs kafka poller.Jun 15 2018, 11:26 PM
Smalyshev added a project: User-Smalyshev.
Smalyshev moved this task from Backlog to Next on the User-Smalyshev board.
Smalyshev added a project: Discovery-Wikidata-Query-Service-Sprint.
Smalyshev moved this task from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.Jul 3 2018, 8:09 AM
gerritbot added a comment.Jul 6 2018, 9:50 PM

Change 421348 merged by jenkins-bot:
[wikidata/query/rdf@master] Store Kafka offsets in the DB

https://gerrit.wikimedia.org/r/421348

Smalyshev moved this task from Next to In review on the User-Smalyshev board.Jul 9 2018, 4:59 PM
Smalyshev moved this task from In progress to Done on the Discovery-Wikidata-Query-Service-Sprint board.Jul 12 2018, 5:34 PM
Smalyshev closed this task as Resolved.Jul 12 2018, 6:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL