{stag} EventLogging on Kafka
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• kevinator
	Jun 12 2015, 7:28 AM

Description

Objective: Move the EventLogging pipeline to use Kafka
Key Result: 10x capacity increate for EventLogging

See Lightning Talk presented October 27th: https://www.youtube.com/watch?v=yUQ5d192z3M
slides are here: https://www.mediawiki.org/wiki/File:EventLogging_on_Kafka_-_Lightning_Talk.pdf

Related Objects
Search...

Status	Assigned	Task
Resolved	• kevinator	T102225 {stag} EventLogging on Kafka
Resolved	Ottomata	T102831 Prep work for Eventlogging on Kafka {stag}
Resolved	Ottomata	T98781 Processor writes valid and invalid events to separate Kafka topics {stag} [13 points]
Resolved	akosiaris	T103381 Open tcp ports 8421 and 8422 to eventlog1001 to the Analytics VLAN
Resolved	Ottomata	T106254 Make EventLogging monitoring and alerts based on Kafka metrics {stag} [8 pts]
Resolved	Ottomata	T111182 Fix auto_offset_reset bug in pykafka and build new .deb package {stag} [8 pts]
Resolved	Ottomata	T106252 Build .deb package for pykafka {stag} [3 pts]
Resolved	Ottomata	T104228 Puppetize parallel eventlogging-processor {stag} [5 pts]
Resolved	Ottomata	T106581 Build 0.8.2.1 Kafka package and upgrade Kafka brokers
Declined	Ottomata	T98161 Build Kafka 0.8.1.1 package for Jessie and upgrade Brokers to Jessie.
Resolved	Ottomata	T103106 Create jmxtrans Jessie package
Resolved	Ottomata	T90640 Audit hyperthreading on analytics nodes.
Resolved	• madhuvishy	T106253 Regularly purge EventLogging data in Hadoop {stag} [8 pts]
Resolved	Ottomata	T106255 Send raw client side events to Kafka using varnishkafka instead of varnishncsa {stag}
Resolved	Ottomata	T106260 Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [13 pts]
Resolved	Ottomata	T98784 Write a new Camus consumer and store the data for Event Logging {stag} [21 pts]
Resolved	• madhuvishy	T104226 Implement parallel eventlogging-processor {stag} [13 pts]
Resolved	Ottomata	T104229 Load test parallel eventlogging-processor {stag} [5 pts]
Declined	• madhuvishy	T109244 Move Eventlogging Kafka writer to use pykafka's Producer instead of python-kafka {stag} [8 pts]
Resolved	Ottomata	T112688 Bug: client IP is being hashed differently by the different parallel processors {stag} [13 pts]

Event Timeline

• kevinator created this task.Jun 12 2015, 7:28 AM

• kevinator raised the priority of this task from to Needs Triage.

• kevinator updated the task description. (Show Details)

• kevinator added projects: Analytics-Kanban, MediaWiki-extensions-EventLogging.

• kevinator subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2015, 7:28 AM

• kevinator triaged this task as Medium priority.Jun 12 2015, 7:32 AM

• kevinator moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.

• kevinator set Security to None.

Ottomata mentioned this in T102831: Prep work for Eventlogging on Kafka {stag}.Jul 19 2015, 5:48 AM

Ottomata added a subtask: T104228: Puppetize parallel eventlogging-processor {stag} [5 pts].Jul 19 2015, 5:51 AM

Ottomata added a subtask: T106256: Kafka Client for MediaWiki.Jul 19 2015, 6:05 AM

Ottomata mentioned this in T106257: Send raw server side events to Kafka using a PHP Kafka Client {oryx}.

Ottomata removed a subtask: T106256: Kafka Client for MediaWiki.Jul 19 2015, 6:07 AM

Ottomata moved this task from Backlog to {stag} EventLogging on Kafka on the MediaWiki-extensions-EventLogging board.Jul 19 2015, 6:11 AM

EventLogging {stag}.jpg (554×727 px, 98 KB)

• kevinator added subtasks: T98781: Processor writes valid and invalid events to separate Kafka topics {stag} [13 points], T98784: Write a new Camus consumer and store the data for Event Logging {stag} [21 pts].Jul 26 2015, 10:52 PM

• kevinator added subtasks: T104226: Implement parallel eventlogging-processor {stag} [13 pts], T104229: Load test parallel eventlogging-processor {stag} [5 pts], T106254: Make EventLogging monitoring and alerts based on Kafka metrics {stag} [8 pts].Jul 26 2015, 10:57 PM

• kevinator closed subtask T106252: Build .deb package for pykafka {stag} [3 pts] as Resolved.Jul 31 2015, 3:22 PM

• kevinator closed subtask T98781: Processor writes valid and invalid events to separate Kafka topics {stag} [13 points] as Resolved.Aug 4 2015, 4:01 PM

• kevinator added a subtask: T109244: Move Eventlogging Kafka writer to use pykafka's Producer instead of python-kafka {stag} [8 pts].Aug 17 2015, 4:57 PM

• madhuvishy closed subtask T109244: Move Eventlogging Kafka writer to use pykafka's Producer instead of python-kafka {stag} [8 pts] as Declined.Aug 25 2015, 3:52 PM

• kevinator closed subtask T104226: Implement parallel eventlogging-processor {stag} [13 pts] as Resolved.Aug 29 2015, 12:11 AM

• kevinator closed subtask T104229: Load test parallel eventlogging-processor {stag} [5 pts] as Resolved.Sep 8 2015, 10:35 PM

• kevinator closed subtask T106254: Make EventLogging monitoring and alerts based on Kafka metrics {stag} [8 pts] as Resolved.Sep 10 2015, 7:54 PM

• kevinator closed subtask T102831: Prep work for Eventlogging on Kafka {stag} as Resolved.

• kevinator closed subtask T106255: Send raw client side events to Kafka using varnishkafka instead of varnishncsa {stag} as Resolved.

Ottomata added a subtask: T110903: Make webperf eventlogging consumers use eventlogging on Kafka.Sep 11 2015, 3:23 PM

• kevinator closed subtask T106260: Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [13 pts] as Resolved.Sep 15 2015, 3:28 PM

• kevinator closed subtask T104228: Puppetize parallel eventlogging-processor {stag} [5 pts] as Resolved.

• kevinator closed subtask T106253: Regularly purge EventLogging data in Hadoop {stag} [8 pts] as Resolved.Sep 25 2015, 3:33 PM

Dan and I just pushed Analytics schema events through this system. During our largest test, we posted around 9000 valid events per second through by using ab to hammer bits.wikimedia.org using 24 cores on 12 different nodes to post.

ab -n 100000 -c 24  'https://bits.wikimedia.org/beacon/event?%7B%22event%22%3A%7B%22name%22%3A%22blah%22%2C%22age%22%3A33%7D%2C%22revision%22%3A13317883%2C%22schema%22%3A%22Analytics%22%2C%22webHost%22%3A%22meta.wikimedia.org%22%2C%22wiki%22%3A%22metawiki%22%7D;'

EventLogging on Kafka with 12 parallelized client side processors seemed to scale fairly linearly. The charts below are a little inaccurate, since they show a 1 minute average, and our test lasted less than 2 minutes. The Analytics schema is a simple use case, and more complicated schemas may take more power to validate, but this test shows overall that we can now scale EventLogging linearly.

Also, since Kafka buffers the raw events, if we do push more events through the system than our current # of processors can process, the events will just remained buffered, but not dropped. A temporary spike of data larger than our bandwidth will only delay processing of events.