Today, we noticed that the flink-app-consumer-cloudelastic taskmanagers were down for about 16 hours. The flink jobmanager marked the job as FAILED, but the flink kubernetes operator did not try to start a new job.
Creating this ticket to stabilize the service. Possible solutions include (but are not limited to):
Change #1017115 had a related patch set uploaded (by Ryan Kemper; author: Bking):
[operations/deployment-charts@master] flink-kubernetes-operator: restart failed jobs
Change #1017296 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] cirrus-streaming-updater: Increase taskManager memory for cloudelastic job
Change #1017296 merged by jenkins-bot:
[operations/deployment-charts@master] cirrus-streaming-updater: Increase taskManager memory for cloudelastic job
The consumer seems generally stable. It involved changes to both the application for better error handling, and an increase in the taskmanager memory above. The pods had been running for a week uninterrupted until we brought them down yesterday to verify some new alerting.
Change #1017115 abandoned by Bking:
[operations/deployment-charts@master] flink-kubernetes-operator: restart failed jobs
Reason:
Not needed to fix problems detailed in ticket; we can always make a new CR if needed in the future
Per @EBernhardson comment above, I'm doing to resolve the ticket. @dcausse or anyone else, if you think we need to look into this some more, feel free to re-open.