Page MenuHomePhabricator

EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. {oryx} [13 pts]
Closed, ResolvedPublic

Description

On kafka metadata change (e.g. broker restart), The EventLogging kafka consumer seems to die and not come back up:

2015-11-10 20:15:36,010 (Thread-13 ) Autocommitting consumer offset for consumer group mysql-m4-master and topic eventlogging-valid-mixed
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/dist-packages/pykafka/simpleconsumer.py", line 269, in fetcher
    self.fetch()
  File "/usr/lib/python2.7/dist-packages/pykafka/simpleconsumer.py", line 500, in fetch
    min_bytes=self._fetch_min_bytes
  File "/usr/lib/python2.7/dist-packages/pykafka/broker.py", line 222, in fetch_messages
    return future.get(FetchResponse)
  File "/usr/lib/python2.7/dist-packages/pykafka/handlers.py", line 55, in get
    raise self.error
SocketDisconnectedError

After this, the offset commit thread continues to run, but no new data is consumed from Kafka.

Event Timeline

Ottomata raised the priority of this task from to High.
Ottomata updated the task description. (Show Details)

I've only noticed this happening for the MySQL consumer, but I'm not sure why it would be different than the others.

We should reproduce this in labs with current version of pykafka, then upgrade, and then see if the problem goes away!

Nuria renamed this task from EventLogging (MySQL?) Kafka consumer stops consuming after Kafka metadata change to EventLogging Kafka consumer stops consuming after Kafka metadata change.Nov 30 2015, 5:48 PM
Nuria renamed this task from EventLogging Kafka consumer stops consuming after Kafka metadata change to EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it..Nov 30 2015, 5:52 PM
Ottomata renamed this task from EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. to EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. [13 pts].Nov 30 2015, 6:04 PM
Milimetric renamed this task from EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. [13 pts] to EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. {oryx} [13 pts].Nov 30 2015, 6:05 PM

I have just reproduced this in labs, and can confirm that upgrading to pykafka 2.1.0 fixes the problem. I will build a .deb and install it in beta for a while to test some more.

Change 257968 had a related patch set uploaded (by Ottomata):
Update to pykafka 2.1.0

https://gerrit.wikimedia.org/r/257968

I have just reproduced this in labs, and can confirm that upgrading to pykafka 2.1.0 fixes the problem. I will build a .deb and install it in beta for a while to test some more.

Excellent!

Change 257968 merged by Ottomata:
Update to pykafka 2.1.0

https://gerrit.wikimedia.org/r/257968