Page MenuHomePhabricator

Incident: EventLogging mysql consumer stopped consuming from kafka {oryx}
Closed, ResolvedPublic

Description

Oct 15, EventLogging mysql consumer gets a StopIteration error and is restarted.
After that, it is running but not consuming from kafka...
After a couple hours, we acknowledge that and restart the consumer.
And it starts consuming all pending events from kafka so quickly that the process is killed because of oom.
Lots of events are lost.

Event Timeline

Milimetric raised the priority of this task from to High.
Milimetric updated the task description. (Show Details)
Milimetric added a project: Analytics-Kanban.
Milimetric added a subscriber: Milimetric.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2015, 10:07 PM
mforns renamed this task from Enable EL consumer to deal with a lot of pressure from kafka {oryx} to Incident: EventLogging mysql consumer stopped consuming from kafka {oryx}.Oct 15 2015, 10:21 PM
mforns assigned this task to Milimetric.
mforns updated the task description. (Show Details)
mforns set Security to None.

Change 246796 had a related patch set uploaded (by Mforns):
Block mysql consumer if the queue is too big

https://gerrit.wikimedia.org/r/246796

Milimetric reassigned this task from Milimetric to mforns.Oct 16 2015, 4:22 PM
Milimetric claimed this task.
Milimetric moved this task from Next Up to In Code Review on the Analytics-Kanban board.

Change 246796 merged by Nuria:
Block mysql consumer if the queue is too big

https://gerrit.wikimedia.org/r/246796

Milimetric moved this task from Ready to Deploy to Paused on the Analytics-Kanban board.
Nuria moved this task from Paused to In Progress on the Analytics-Kanban board.Oct 26 2015, 4:15 PM
mforns claimed this task.Oct 26 2015, 5:09 PM
mforns removed a project: Patch-For-Review.

Executing the backfilling right now.
Seems to be working fine, will take couple hours.

mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
mforns moved this task from Done to Ready to Deploy on the Analytics-Kanban board.
mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Oct 27 2015, 4:49 PM
Nuria closed this task as Resolved.Oct 28 2015, 2:47 PM