During the course of deploying Zookeeper in T344614 , I left the service down too long and we lost the ability to restore from a checkpoint.
Fixing this requires bootstrapping a new savepoint , a process that is a bit ad-hoc at the moment.
Creating this ticket to:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T317045 [Epic] Re-architect the Search Update Pipeline | |||
| Open | None | T340548 [EPIC] Deployment of the Search Update Pipeline on Flink / k8s | |||
| Open | lbowmaker | T328561 Flink Operations | |||
| Open | bking | T326409 Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model | |||
| Open | bking | T342149 Test common operations in the flink operator/k8s/Flink ZK environment | |||
| Resolved | bking | T344614 Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster | |||
| Resolved | bking | T345957 Restore dse-k8s' rdf-streaming-updater from savepoint/improve bootstrapping process | |||
| Resolved | bking | T346048 Troubleshoot rdf-streaming-updater/dse-k8s cluster |
Thanks to @dcausse , the job has recovered and we've updated the docs. Moving to "Needs Review" so he can confirm the health of flink-app in dse-k8s and review the docs.
@bking thanks! I can confirm that the job is running fine, the dashboards show some activity the test stream wdqs_streaming_updater_test_T289836 is seeing all the mutations.
Regarding H/A with zookeeper I believe it's properly using as I don't see the usual k8s configmaps being created when the KUBERNETES H/A mode was used. I believe we should be good to do some testing of the various flink operations.
Thanks for all your help as well. I believe this is done, but please let us know if we need to revisit the docs or restore process.