Page MenuHomePhabricator

Create LimitedConcurrencyKafkaConsumer
Open, LowPublic

Description

Currently the BaseExecutor in ChangeProp is overloaded with code. It became too complex and needs refactoring.

One issue is that it's simultaneously controlling event fetching from Kafka and a lot of the application logic. The first thing to extract from it is event fetching. We can not just use the normal KafkaConsumer flowing mode, because it fetches events as fast as possible and will quickly fill up the memory with backlog events. What we need is a LimitedConcurrencyKafkaConsumer - it will only allow a certain number of events being 'in process' and will control when to fetch the next one.

I'm not sure yet where to draw the line on the API surface for it, and what API do we want it to expose, but this will help simplify the BaseExecutor class.