Large message causes our kafka implementation to fail in various places:
- on the daemon, because python-kafka applies its message size prior to message compression
- Increase python-kafka max request size to 40MB because our messages compress very well.
- on the spark workers using KafkaUtil.createRDD: Ran out of messages before reaching ending offset 182100 for topic mjolnir_result partition 0 start 152128. This should not happen, and indicates that messages may have been lost.
- it's probably that we need to tweak by adding some config params to KafkaUtils.createRDD.