re-enable wdqs kafka poller
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gehel
	Mar 12 2018, 10:27 AM

Description

WDQS Kafka Poller has been enabled first at part of T188252. It was disabled after some mirroring issues between kafka-main and kafka-jumbo. I tried to re-enable it this morning, but we had a different mirroring issue. WDQS is now back on the RC updater, and works fine.

We should re-enable the kafka poller, but only after the different mirroring issues are solved (I'm not entirely clear what those are, but @elukey will document them in a ticket, which we'll link from here).

Details

	Subject	Repo	Branch	Lines +/-
	Switch public cluster to Kafka event source	operations/puppet	production	+2 -2
	Enable kafka poller on test hosts	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Gehel	T189458 re-enable wdqs kafka poller
Resolved	Ottomata	T189464 Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo
Resolved	Ottomata	T189611 Alert for Kafka MirrorMaker lag
Resolved	Ottomata	T190049 Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent Replicator
Resolved	Smalyshev	T192963 Store Kafka poller position data in the WDQS database

Event Timeline

Gehel created this task.Mar 12 2018, 10:27 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2018, 10:27 AM

Gehel added a subtask: T189464: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo.Mar 12 2018, 2:11 PM

Smalyshev added a project: Wikidata-Query-Service.Mar 12 2018, 5:08 PM

Restricted Application added projects: Wikidata, Discovery-ARCHIVED. · View Herald TranscriptMar 12 2018, 5:08 PM

Smalyshev triaged this task as High priority.Mar 12 2018, 6:27 PM

Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Mar 13 2018, 4:39 PM

Gehel claimed this task.Mar 13 2018, 5:27 PM

Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Apr 7 2018, 1:02 AM

Smalyshev edited projects, added User-Smalyshev; removed Discovery-Wikidata-Query-Service-Sprint.Apr 24 2018, 9:59 PM

Smalyshev moved this task from Current work to Incoming on the Wikidata-Query-Service board.Apr 26 2018, 5:44 PM

• Jonas subscribed.Jun 5 2018, 12:43 PM

• Nuria closed subtask T189464: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo as Resolved.Jun 11 2018, 11:04 PM

Smalyshev added a subtask: T192963: Store Kafka poller position data in the WDQS database.Jun 15 2018, 11:26 PM

Smalyshev closed subtask T192963: Store Kafka poller position data in the WDQS database as Resolved.Jul 12 2018, 6:48 PM

Current plan is to try again after July 23th, test hosts first and then others.

Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Jul 27 2018, 9:04 PM

Change 449109 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Revert "Revert "Enable kafka poller on test hosts""

https://gerrit.wikimedia.org/r/449109

gerritbot added a project: Patch-For-Review.Jul 30 2018, 4:46 AM

• Jonas mentioned this in T192567: Expose constraint violations to WDQS.Aug 1 2018, 11:14 AM

• Jonas mentioned this in T201147: Expose constraint violations to WDQS using event queue.Aug 3 2018, 9:08 AM

In T189458#4420797, @Smalyshev wrote:

Current plan is to try again after July 23th, test hosts first and then others.

Is this still the current plan? Any ETA? Asking because it is needed for T201147.

Change 449109 merged by Gehel:
[operations/puppet@production] Enable kafka poller on test hosts

https://gerrit.wikimedia.org/r/449109

Smalyshev moved this task from Next to Doing on the User-Smalyshev board.Sep 10 2018, 8:57 PM

Smalyshev moved this task from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.Sep 12 2018, 5:03 AM

Change 462907 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] Switch public cluster to Kafka event source

https://gerrit.wikimedia.org/r/462907

Smalyshev moved this task from Doing to In review on the User-Smalyshev board.Oct 1 2018, 7:35 AM

Change 462907 merged by Gehel:
[operations/puppet@production] Switch public cluster to Kafka event source

https://gerrit.wikimedia.org/r/462907

Gehel moved this task from In progress to Done on the Discovery-Wikidata-Query-Service-Sprint board.Oct 2 2018, 5:27 PM

Smalyshev moved this task from In review to Done on the User-Smalyshev board.Oct 2 2018, 5:44 PM

Smalyshev closed this task as Resolved.Oct 3 2018, 7:58 PM