Page MenuHomePhabricator

Create kafka-based recent change poller
Closed, ResolvedPublic

Description

Based on T161731, we want to create a recent change poller that would get information from Kafka change stream instead of RecentChanges API. This would allow to get recent changes from many clients without increasing load on Wikidata and would make change fetching more reliable.

Event Timeline

Restricted Application added projects: Wikidata, Discovery. · View Herald TranscriptJan 29 2018, 9:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev triaged this task as Medium priority.Jan 30 2018, 8:05 AM

Change 406958 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] [WIP] Add Kafka poller

https://gerrit.wikimedia.org/r/406958

Change 412873 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: allow configuration of kafka based updates

https://gerrit.wikimedia.org/r/412873

elukey added a subscriber: elukey.Feb 21 2018, 10:08 AM

Change 406958 merged by jenkins-bot:
[wikidata/query/rdf@master] Add Kafka poller

https://gerrit.wikimedia.org/r/406958

Change 412873 merged by Gehel:
[operations/puppet@production] wdqs: allow configuration of kafka based updates

https://gerrit.wikimedia.org/r/412873

Smalyshev closed this task as Resolved.Mar 1 2018, 7:50 PM