Large message causes our kafka implementation to fail in various places:

• on the daemon, because python-kafka applies its message size prior to message compression
  • Increase python-kafka max request size to 40MB because our messages compress very well.
• on the spark workers using KafkaUtil.createRDD: Ran out of messages before reaching ending offset 182100 for topic mjolnir_result partition 0 start 152128. This should not happen, and indicates that messages may have been lost.
  • it's probably that we need to tweak by adding some config params to KafkaUtils.createRDD.