When enriched event results in payloads larger than max.request.size, Flink's Kafka Sink producer will fail with org.apache.kafka.common.errors.RecordTooLargeException.
This will result in the message getting lost and the Taskmanager shutting down, and potentially failing to restart.
To improve application reliability, we should filter out these messages before producing them and forward the origin event to the application's error topic.
Related work:
We did implement this filter in the old Scala PoC: https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-stream-enrichment/-/blob/main/enrichment/src/main/scala/org/wikimedia/mediawiki/event/enrichment/Enrichment.scala#L185
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/69
Draft: Don't process event with large content body.
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/69
Don't process events with large content body.