Page MenuHomePhabricator

Incident: EventLogging mysql consumer stopped consuming from kafka {oryx}
Closed, ResolvedPublic

Description

Oct 15, EventLogging mysql consumer gets a StopIteration error and is restarted.
After that, it is running but not consuming from kafka...
After a couple hours, we acknowledge that and restart the consumer.
And it starts consuming all pending events from kafka so quickly that the process is killed because of oom.
Lots of events are lost.

Event Timeline

Milimetric raised the priority of this task from to High.
Milimetric updated the task description. (Show Details)
Milimetric added a project: Analytics-Kanban.
Milimetric subscribed.
mforns renamed this task from Enable EL consumer to deal with a lot of pressure from kafka {oryx} to Incident: EventLogging mysql consumer stopped consuming from kafka {oryx}.Oct 15 2015, 10:21 PM
mforns assigned this task to Milimetric.
mforns updated the task description. (Show Details)
mforns set Security to None.

Change 246796 had a related patch set uploaded (by Mforns):
Block mysql consumer if the queue is too big

https://gerrit.wikimedia.org/r/246796

Milimetric claimed this task.
Milimetric moved this task from Next Up to In Code Review on the Analytics-Kanban board.

Change 246796 merged by Nuria:
Block mysql consumer if the queue is too big

https://gerrit.wikimedia.org/r/246796

mforns removed a project: Patch-For-Review.

Executing the backfilling right now.
Seems to be working fine, will take couple hours.

mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
mforns moved this task from Done to Ready to Deploy on the Analytics-Kanban board.