Page MenuHomePhabricator

Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [13 pts]
Closed, ResolvedPublic

Description

  • Stop puppet on eventlog1001
  • Deploy latest EL code to eventlog1001

Changes to configs and daemons:

  • server-side forwarder: UDP -> Kafka eventlogging-server-side, restart forwader.
  • remove client-side forwarder. raw client side events now come from varnishkafka
  • client-side and server-side processors consume from Kafka topics and produce to schema based topics & eventlogging-all & eventlogging_EventError
  • remove multiplexer.
  • remove reporters
  • remove graphite consumer from hafnium
  • remove client-side and server-side log consumers
  • all-events-log consumers from eventlogging-all
  • mysql-m4-master consumes from eventlogging-all

Event Timeline

Ottomata created this task.Jul 19 2015, 7:07 AM
Ottomata raised the priority of this task from to Normal.
Ottomata updated the task description. (Show Details)
Ottomata added subscribers: kevinator, Aklapper, Ottomata.
Ottomata set Security to None.
kevinator renamed this task from Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} to Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [3 pts].Aug 17 2015, 4:53 PM
kevinator moved this task from Incoming to Tasked on the Analytics-Backlog board.

Change 236853 had a related patch set uploaded (by Ottomata):
Make EventLogging MySQL consumer consume from kafka instead of 0mq

https://gerrit.wikimedia.org/r/236853

Change 236853 merged by Ottomata:
Puppetize EventLogging on Kafka server-side-forwarder on eventlog1001

https://gerrit.wikimedia.org/r/236853

Change 236921 had a related patch set uploaded (by Ottomata):
Make eventlogging server side processor consume from and produce to kafka

https://gerrit.wikimedia.org/r/236921

Change 236921 merged by Ottomata:
Make eventlogging server side processor consume from and produce to kafka

https://gerrit.wikimedia.org/r/236921

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 237249 had a related patch set uploaded (by Ottomata):
EventLogging client side processor now consumes from Kafka

https://gerrit.wikimedia.org/r/237249

Ottomata claimed this task.Sep 9 2015, 8:22 PM

Change 237249 merged by Ottomata:
EventLogging client side processor now consumes from Kafka

https://gerrit.wikimedia.org/r/237249

Change 237255 had a related patch set uploaded (by Ottomata):
Leave eventlogging server side raw forwarder outputting to zmq until we are ready to turn off all zmq

https://gerrit.wikimedia.org/r/237255

Change 237255 merged by Ottomata:
Leave eventlogging server side raw forwarder outputting to zmq until we are ready to turn off all zmq

https://gerrit.wikimedia.org/r/237255

Ottomata renamed this task from Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [3 pts] to Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [8 pts].Sep 9 2015, 8:53 PM

Change 237261 had a related patch set uploaded (by Ottomata):
Make eventlogging files consumer consume from Kafka instead of ZMQ

https://gerrit.wikimedia.org/r/237261

Change 237261 merged by Ottomata:
Make eventlogging files consumer consume from Kafka instead of ZMQ

https://gerrit.wikimedia.org/r/237261

Change 237469 had a related patch set uploaded (by Ottomata):
Make eventlogging mysql consumer consume from kafka instead of ZMQ

https://gerrit.wikimedia.org/r/237469

Change 237469 merged by Ottomata:
Make eventlogging mysql consumer consume from kafka instead of ZMQ

https://gerrit.wikimedia.org/r/237469

Change 237479 had a related patch set uploaded (by Ottomata):
Remove eventlogging multiplexer, forward kafka eventlogging events to zmq port 8600

https://gerrit.wikimedia.org/r/237479

Change 237479 merged by Ottomata:
Remove eventlogging multiplexer, forward kafka eventlogging events to zmq port 8600

https://gerrit.wikimedia.org/r/237479

Change 237490 had a related patch set uploaded (by Ottomata):
Turn off varnishncsa udp logging for eventlogging

https://gerrit.wikimedia.org/r/237490

Change 237490 merged by Ottomata:
Turn off varnishncsa udp logging for eventlogging

https://gerrit.wikimedia.org/r/237490

ggellerman renamed this task from Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [8 pts] to Deploy EventLogging on Kafka to eventlog1001 (aka production!) {stag} [13 pts].Sep 11 2015, 3:36 PM
kevinator closed this task as Resolved.Sep 15 2015, 3:28 PM