Page MenuHomePhabricator

Prep work for Eventlogging on Kafka {stag}
Closed, ResolvedPublic

Related Objects

Event Timeline

Ottomata created this task.Jun 17 2015, 3:49 PM
Ottomata claimed this task.
Ottomata raised the priority of this task from to Normal.
Ottomata updated the task description. (Show Details)
kevinator renamed this task from Prep work for Eventlogging on Kafka to Prep work for Eventlogging on Kafka {stag}.Jun 18 2015, 3:43 PM
kevinator set Security to None.

Change 220912 had a related patch set uploaded (by Ottomata):
Refactor eventlogging role classes to make it easier to include different processes on different hosts

https://gerrit.wikimedia.org/r/220912

Change 220912 merged by Ottomata:
Refactor eventlogging role classes to make it easier to include different processes on different hosts

https://gerrit.wikimedia.org/r/220912

Change 221145 had a related patch set uploaded (by Ottomata):
Configure eventlogging kafka forwarder and processor

https://gerrit.wikimedia.org/r/221145

Change 221145 merged by Ottomata:
Configure eventlogging kafka forwarder and processor

https://gerrit.wikimedia.org/r/221145

So, there are two high level {stag} Kafka on EventLogging tickets. This one, and T102225. I think there are a few more prep things that need to be done before we can close this one and focus on the other.

  • Review and merge Madhu's BalancedConsumer patch in T104226
  • Increment EventLogging version.
  • Load test BalancedConsumer on analytics1010 T104229 with real data.
  • Run eventlogging in test mode on analytics1010 for a while

Once eventlogging is running on analytics1010 and things look good, I think we can close this ticket and proceed with tasks to finalize {stag} in T102225.

We want to use pykafka to use BalancedConsumer. This client requires a later version of Kafka than we have. Now we have to figure out how to package it :(

Ottomata moved this task from Paused to In Progress on the Analytics-Kanban board.Aug 28 2015, 3:50 PM
Ottomata moved this task from Paused to Done on the Analytics-Kanban board.Sep 9 2015, 8:22 PM