Description

Based on T161731, we want to create a recent change poller that would get information from Kafka change stream instead of RecentChanges API. This would allow to get recent changes from many clients without increasing load on Wikidata and would make change fetching more reliable.

Details

	Subject	Repo	Branch	Lines +/-
	wdqs: allow configuration of kafka based updates	operations/puppet	production	+23 -7
	Add Kafka poller	wikidata/query/rdf	master	+1 K -32

Customize query in gerrit

Related Objects

Mentioned In: T188252: Activate kafka-based recent change poller for wikidata query service
Mentioned Here: T161731: Create reliable change stream for specific wiki

Event Timeline

Smalyshev created this task.Jan 29 2018, 9:56 PM

Restricted Application added projects: Wikidata, Discovery-ARCHIVED. · View Herald TranscriptJan 29 2018, 9:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Smalyshev added projects: User-Smalyshev, Discovery-Wikidata-Query-Service-Sprint.Jan 29 2018, 11:47 PM

Smalyshev moved this task from Backlog to Doing on the User-Smalyshev board.

Smalyshev triaged this task as Medium priority.Jan 30 2018, 8:05 AM

Smalyshev moved this task from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.Jan 30 2018, 6:18 PM

Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Jan 30 2018, 11:11 PM

Change 406958 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] [WIP] Add Kafka poller

https://gerrit.wikimedia.org/r/406958

gerritbot added a project: Patch-For-Review.Jan 30 2018, 11:16 PM

Smalyshev moved this task from In progress to Needs review on the Discovery-Wikidata-Query-Service-Sprint board.Feb 17 2018, 1:31 AM

Smalyshev moved this task from Doing to In review on the User-Smalyshev board.

Change 412873 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: allow configuration of kafka based updates

https://gerrit.wikimedia.org/r/412873

elukey subscribed.Feb 21 2018, 10:08 AM

Change 406958 merged by jenkins-bot:
[wikidata/query/rdf@master] Add Kafka poller

https://gerrit.wikimedia.org/r/406958

Smalyshev moved this task from Needs review to Done on the Discovery-Wikidata-Query-Service-Sprint board.Feb 22 2018, 12:53 AM

Gehel mentioned this in T188252: Activate kafka-based recent change poller for wikidata query service.Feb 26 2018, 2:26 PM

Change 412873 merged by Gehel:
[operations/puppet@production] wdqs: allow configuration of kafka based updates

https://gerrit.wikimedia.org/r/412873

Smalyshev closed this task as Resolved.Mar 1 2018, 7:50 PM