When mixing incoming streams with different rate (e.g. one high volume stream and another with rare messages) the extractor BoundedOutOfOrdernessTimestampExtractor we use is not designed to deal with such configuration as it requires continuous flow on the stream to emit "sane" watermarks.
In the case a stream is not emitting messages for long the whole pipeline might be stuck just waiting for a message from that stream to trigger the next lower watermark. The reason is that the default watermark generators in flink do not take the responsibility to decide if the stream is simply late or inactive.
There seems to be two solutions to address this problem:
- have a watermark generator that also inspect the processing time and emits a watermark with processing time - Xsec if no events have been seen in a while, this will unblock downstream operators like windows function waiting for low watermark for firing the windows that started to accumulate.
- Investigate using org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext#markAsTemporarilyIdle (wrap FlinkKafkaConsumer ?)
- when adding a low rate stream the operators relying on low watermarks must not be waiting for events from this stream to trigger their operations.