⚓ T98781 Processor writes valid and invalid events to separate Kafka topics {stag} [13 points]

	Subject	Repo	Branch	Lines +/-
	Modify eventlogging module to use new changes to eventlogging server	operations/puppet	production	+42 -64
	Add optional output for invalid events	mediawiki/extensions/EventLogging	master	+14 -0

Status	Assigned	Task
Resolved	• kevinator	T102225 {stag} EventLogging on Kafka
Resolved	Ottomata	T102831 Prep work for Eventlogging on Kafka {stag}
Resolved	Ottomata	T98781 Processor writes valid and invalid events to separate Kafka topics {stag} [13 points]
Resolved	akosiaris	T103381 Open tcp ports 8421 and 8422 to eventlog1001 to the Analytics VLAN
Resolved	Ottomata	T106254 Make EventLogging monitoring and alerts based on Kafka metrics {stag} [8 pts]
Resolved	Ottomata	T106581 Build 0.8.2.1 Kafka package and upgrade Kafka brokers
Declined	Ottomata	T98161 Build Kafka 0.8.1.1 package for Jessie and upgrade Brokers to Jessie.
Resolved	Ottomata	T103106 Create jmxtrans Jessie package
Resolved	Ottomata	T90640 Audit hyperthreading on analytics nodes.
Resolved	Ottomata	T111182 Fix auto_offset_reset bug in pykafka and build new .deb package {stag} [8 pts]

• ggellerman created this task.May 11 2015, 9:19 PM

• ggellerman assigned this task to Milimetric.

• ggellerman raised the priority of this task from to Needs Triage.

• ggellerman updated the task description. (Show Details)

• ggellerman added subscribers: • ggellerman, • kevinator, mforns.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2015, 9:19 PM

Milimetric renamed this task from Code to change the Processor to write invalid events to a separate Kafka topic from valid events [13 points] to Processor writes valid and invalid events to separate Kafka topics [13 points] {oryx}.May 12 2015, 3:46 PM

Milimetric set Security to None.

Milimetric added a project: Analytics-Kanban.

Milimetric moved this task from Next Up to In Progress on the Analytics-Kanban board.

Progress so far - I have a code change that should work but I don't know how to use kafka on beta labs. Zookeeper seems to not be running there.

Change 210729 had a related patch set uploaded (by Milimetric):
Add optional output for invalid events

https://gerrit.wikimedia.org/r/210729

gerritbot added a project: Patch-For-Review.May 13 2015, 6:09 PM

Change 210765 had a related patch set uploaded (by Milimetric):
[WIP] Add parallel kafka pipeline

https://gerrit.wikimedia.org/r/210765

Milimetric moved this task from In Progress to In Code Review on the Analytics-Kanban board.May 29 2015, 2:00 PM

• ggellerman moved this task from In Code Review to Paused on the Analytics-Kanban board.Jun 5 2015, 3:37 PM

Change 210729 merged by Ottomata:
Add optional output for invalid events

https://gerrit.wikimedia.org/r/210729

Ottomata mentioned this in rEEVL945ffcfb31f3: Add optional output for invalid events.Jun 5 2015, 9:02 PM

Milimetric mentioned this in rMEXT84f032cdf05d: Updated mediawiki/extensions Project: mediawiki/extensions/EventLogging….Jun 5 2015, 9:03 PM

• ggellerman reassigned this task from Milimetric to Ottomata.Jun 8 2015, 3:34 PM

• ggellerman moved this task from Paused to In Progress on the Analytics-Kanban board.

• ggellerman added a subscriber: Milimetric.

https://gerrit.wikimedia.org/r/210729 (branch master): WMF-deploy-2015-06-09_(1.26wmf9)

• kevinator renamed this task from Processor writes valid and invalid events to separate Kafka topics [13 points] {oryx} to Processor writes valid and invalid events to separate Kafka topics [13 points] {stag}.Jun 8 2015, 10:24 PM

• ggellerman moved this task from In Progress to Paused on the Analytics-Kanban board.Jun 10 2015, 3:39 PM

• ggellerman moved this task from Paused to In Progress on the Analytics-Kanban board.Jun 17 2015, 3:45 PM

Ottomata added a parent task: T102831: Prep work for Eventlogging on Kafka {stag}.Jun 17 2015, 3:50 PM

Change 210765 merged by Ottomata:
Modify eventlogging module to use new changes to eventlogging server

https://gerrit.wikimedia.org/r/210765

Ottomata mentioned this in rOPUP872b4384e3e7: Modify eventlogging module to use new changes to eventlogging server.Jun 22 2015, 4:06 PM

• kevinator renamed this task from Processor writes valid and invalid events to separate Kafka topics [13 points] {stag} to Processor writes valid and invalid events to separate Kafka topics {stag} [13 points].Jun 24 2015, 3:40 PM

Legoktm removed a subscriber: • Forrestbot.Jun 29 2015, 5:49 PM

• kevinator triaged this task as Medium priority.Jul 2 2015, 3:37 PM

• ggellerman moved this task from In Progress to Paused on the Analytics-Kanban board.Jul 8 2015, 3:35 PM

• kevinator added a parent task: T102225: {stag} EventLogging on Kafka.Jul 26 2015, 10:52 PM

• ggellerman moved this task from Paused to Ready to Deploy on the Analytics-Kanban board.Aug 4 2015, 4:00 PM

• ggellerman moved this task from Ready to Deploy to Done on the Analytics-Kanban board.

work for this task is done and deployed... but code not being used right now. Marking as done as we have moved on.

• kevinator closed this task as Resolved.Aug 4 2015, 4:01 PM

Processor writes valid and invalid events to separate Kafka topics {stag} [13 points]
Closed, ResolvedPublic
Actions

Description

Details

Related Objects
Search...

Event Timeline

Processor writes valid and invalid events to separate Kafka topics {stag} [13 points]Closed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Processor writes valid and invalid events to separate Kafka topics {stag} [13 points]
Closed, ResolvedPublic
Actions

Related Objects
Search...