Page MenuHomePhabricator

Configure Kafka in beta labs + Forwarder modification to produce to multiple output [16 points] {oryx}
Closed, ResolvedPublic

Description

Code for Python & Kafka working in Beta Labs.

Joseph to pair with Marcel

This task includes:
Configure a new Event Logging Processor to read/write using Kafka instead of zmq (write server and client side events to the same topic, no need for multiplexer)
Configure new Event Logging Log Consumers to read/write using Kafka
Change the Mysql Consumer to read from Kafka
Forwarder modification to produce to multiple outputs - 1 to zmq and 1 to kafka

8 of 16 points on this task are borrowed from the task merged in.

Event Timeline

ggellerman assigned this task to JAllemandou.
ggellerman raised the priority of this task from to Needs Triage.
ggellerman updated the task description. (Show Details)
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2015, 9:14 PM
ggellerman renamed this task from Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka to Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points].May 11 2015, 9:15 PM
ggellerman reassigned this task from JAllemandou to mforns.
ggellerman set Security to None.
ggellerman edited subscribers, added: JAllemandou; removed: mforns.

Change 210765 had a related patch set uploaded (by Milimetric):
[WIP] Add parallel kafka pipeline

https://gerrit.wikimedia.org/r/210765

Milimetric renamed this task from Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points] to Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points] {oryx}.May 19 2015, 4:19 PM
madhuvishy renamed this task from Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points] {oryx} to Configure Kafka in beta labs + Forwarder modification to produce to multiple output [16 points] {oryx}.May 28 2015, 11:16 PM
madhuvishy updated the task description. (Show Details)
madhuvishy added a comment.EditedMay 28 2015, 11:45 PM

The python code changes for the forwarder to produce multiple outputs (zmq and kafka) is here - https://gerrit.wikimedia.org/r/#/c/210701/

Change 210701 had a related patch set uploaded (by Madhuvishy):
Modify forwarder to produce multiple outputs

https://gerrit.wikimedia.org/r/210701

This comment was removed by kevinator.

Change 210701 merged by Ottomata:
Modify forwarder to produce multiple outputs

https://gerrit.wikimedia.org/r/210701

ggellerman moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
kevinator closed this task as Resolved.Jun 8 2015, 6:41 PM

Code has been merged, resolving task.

Change 210765 merged by Ottomata:
Modify eventlogging module to use new changes to eventlogging server

https://gerrit.wikimedia.org/r/210765