While attempting to restore a savepoint in T345957, we noticed that the rdf-streaming-updater application hangs. Further exploration reveals that while the k8s workers themselves can resolve DNS, containers running on these workers cannot.
Creating this ticket to troubleshoot and (hopefully) fix this issue.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T317045 [Epic] Re-architect the Search Update Pipeline | |||
| Open | None | T340548 [EPIC] Deployment of the Search Update Pipeline on Flink / k8s | |||
| Open | lbowmaker | T328561 [Event Platform] Flink Operations | |||
| Resolved | bking | T326409 Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model | |||
| Resolved | bking | T342149 Test common operations in the flink operator/k8s/Flink ZK environment | |||
| Resolved | bking | T344614 Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster | |||
| Resolved | bking | T345957 Restore dse-k8s' rdf-streaming-updater from savepoint/improve bootstrapping process | |||
| Resolved | bking | T346048 Troubleshoot rdf-streaming-updater/dse-k8s cluster |
Update: After some consultation in #wikimedia-k8s-sig , this doesn't seem to be a DNS issue. So it's most likely firewall rules...will continue troubleshooting and get back.
Change 956474 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] rdf-streaming-updater-k8s: Add egress rules to values
Change 955032 had a related patch set uploaded (by Bking; author: Ebernhardson):
[operations/deployment-charts@master] Add a networkpolicy template for zookeeper
Change 955032 merged by jenkins-bot:
[operations/deployment-charts@master] Add a networkpolicy template for zookeeper
Change 956474 merged by jenkins-bot:
[operations/deployment-charts@master] rdf-streaming-updater-k8s: Add egress/proxy rules to values
Patch above fixed the firewall rules, and we were able to get the flink-app to restore from savepoint. Closing this, but work continues in T345957 .