Page MenuHomePhabricator
Create Task
Maniphest T98779

Configure Kafka in beta labs + Forwarder modification to produce to multiple output [16 points] {oryx}
Closed, ResolvedPublic
Actions

Description

Code for Python & Kafka working in Beta Labs.

Joseph to pair with Marcel

This task includes:
Configure a new Event Logging Processor to read/write using Kafka instead of zmq (write server and client side events to the same topic, no need for multiplexer)
Configure new Event Logging Log Consumers to read/write using Kafka
Change the Mysql Consumer to read from Kafka
Forwarder modification to produce to multiple outputs - 1 to zmq and 1 to kafka

8 of 16 points on this task are borrowed from the task merged in.

Details

SubjectRepoBranchLines +/-
Modify eventlogging module to use new changes to eventlogging serveroperations/puppetproduction+42 -64
Modify forwarder to produce multiple outputsmediawiki/extensions/EventLoggingmaster+42 -20
Customize query in gerrit

Related Objects

Event Timeline

• ggellerman created this task.May 11 2015, 9:14 PM
• ggellerman assigned this task to JAllemandou.
• ggellerman raised the priority of this task from to Needs Triage.
• ggellerman updated the task description. (Show Details)
• ggellerman added projects: Analytics-Kanban, MediaWiki-extensions-EventLogging.
• ggellerman added subscribers: • ggellerman, • kevinator, mforns.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2015, 9:14 PM
• ggellerman renamed this task from Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka to Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points].May 11 2015, 9:15 PM
• ggellerman reassigned this task from JAllemandou to mforns.
• ggellerman set Security to None.
• ggellerman edited subscribers, added: JAllemandou; removed: mforns.
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.May 12 2015, 3:41 PM
gerritbot subscribed.May 13 2015, 8:02 PM

Change 210765 had a related patch set uploaded (by Milimetric):
[WIP] Add parallel kafka pipeline

https://gerrit.wikimedia.org/r/210765

gerritbot added a project: Patch-For-Review.May 13 2015, 8:02 PM
mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.May 14 2015, 3:38 PM
Milimetric renamed this task from Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points] to Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points] {oryx}.May 19 2015, 4:19 PM
• madhuvishy merged a task: T98780: Configuration for Python & Kafka in Beta labs [8 points] {oryx}.May 28 2015, 11:00 PM
• madhuvishy added subscribers: Milimetric, • madhuvishy.

T98779 and T98780 are covered in the same patch - https://gerrit.wikimedia.org/r/210765. So I'm merging the other configuration task into this one.

• madhuvishy renamed this task from Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points] {oryx} to Configure Kafka in beta labs + Forwarder modification to produce to multiple output [16 points] {oryx}.May 28 2015, 11:16 PM
• madhuvishy updated the task description. (Show Details)
• madhuvishy added a comment.EditedMay 28 2015, 11:45 PM

The python code changes for the forwarder to produce multiple outputs (zmq and kafka) is here - https://gerrit.wikimedia.org/r/#/c/210701/

gerritbot added a comment.May 29 2015, 2:26 AM

Change 210701 had a related patch set uploaded (by Madhuvishy):
Modify forwarder to produce multiple outputs

https://gerrit.wikimedia.org/r/210701

• ggellerman moved this task from In Code Review to Paused on the Analytics-Kanban board.Jun 5 2015, 3:32 PM
• kevinator added a comment.Jun 5 2015, 3:38 PM
This comment was removed by • kevinator.
gerritbot added a comment.Jun 5 2015, 6:10 PM

Change 210701 merged by Ottomata:
Modify forwarder to produce multiple outputs

https://gerrit.wikimedia.org/r/210701

JAllemandou mentioned this in rMEXT1c9775f3b0d1: Updated mediawiki/extensions Project: mediawiki/extensions/EventLogging….Jun 5 2015, 6:10 PM
Ottomata mentioned this in rEEVL81788edb83c8: Modify forwarder to produce multiple outputs.Jun 5 2015, 6:14 PM
• ggellerman moved this task from Paused to Ready to Deploy on the Analytics-Kanban board.Jun 8 2015, 3:45 PM
• ggellerman moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
• Forrestbot added a project: WMF-deploy-2015-06-09_(1.26wmf9).Jun 8 2015, 5:30 PM
• Forrestbot subscribed.
• kevinator closed this task as Resolved.Jun 8 2015, 6:41 PM

Code has been merged, resolving task.

gerritbot added a comment.Jun 22 2015, 4:05 PM

Change 210765 merged by Ottomata:
Modify eventlogging module to use new changes to eventlogging server

https://gerrit.wikimedia.org/r/210765

Ottomata mentioned this in rOPUP872b4384e3e7: Modify eventlogging module to use new changes to eventlogging server.Jun 22 2015, 4:06 PM
Legoktm removed a subscriber: • Forrestbot.Jun 29 2015, 5:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits