Flink ZK environment
Open, In Progress, HighPublic
Actions

Assigned To

Authored By

	bking
	Jul 18 2023, 5:44 PM

Description

Per T328561 , we need to consider and document the different aspects of operating a Flink application (for example: how to handle restarts, upgrades, etc) in the new flink operator/k8s/Flink ZK environment we are currently building .

AC:

Perform/document relevant operations already identified by @gmodena.
How to handle restarting a Flink application ( ref https://phabricator.wikimedia.org/T328563 )
~~Automate Replay of Failed Events (ref https://phabricator.wikimedia.org/T328565 )~~ Doesn't seem relevant to us
Handle app upgrades ( ref https://phabricator.wikimedia.org/T328569 )

Anything that's already done today:

Initial deployment of the Flink job
Version upgrade of the Flink job
Restart the jobs without upgrading: ran kubectl rollout restart deployment flink-app-wdqs on dse-k8s, the Flink jobmanager was able to recover without human intervention.
Recovery on Flink failure (restarts in the same place). We have successfully restored from both checkpoints and savepoints. Documented here
~~Test running multiple flink apps controlled by the same flink operator, using the same Zookeeper instance for Flink HA.~~ Crossing out as this doesn't seem to be a blocker; we have confirmation from the Flink mailing list and personal observation that the Flink Kubernetes Operator uses the name of the helm deployment as the cluster id, so namespace collision shouldn't be a problem.
Stop the operator and see what happens to the application.

Details

Subject	Repo	Branch	Lines +/-
rdf-streaming-updater: take a savepoint manually	operations/deployment-charts	master	+1 -1
dse-k8s: Trigger flink-app savepoint	operations/deployment-charts	master	+1 -1
dse-k8s: Trigger flink-app savepoint	operations/deployment-charts	master	+1 -1
dse-k8s: Manually restore flink-app	operations/deployment-charts	master	+1 -1
dse-k8s: Trigger flink-app savepoint	operations/deployment-charts	master	+5 -0
rdf-streaming-updater: start adding per-env ZK path root	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	bking	T326409 Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model
In Progress	bking	T342149 Test common operations in the flink operator/k8s/Flink ZK environment
Resolved	bking	T344614 Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster
Resolved	bking	T345957 Restore dse-k8s' rdf-streaming-updater from savepoint/improve bootstrapping process
Resolved	bking	T346048 Troubleshoot rdf-streaming-updater/dse-k8s cluster
Resolved	dcausse	T326914 Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink

Event Timeline

bking created this task.Jul 18 2023, 5:44 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptJul 18 2023, 5:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Gehel added a parent task: T317045: [Epic] Re-architect the Search Update Pipeline.Jul 24 2023, 3:29 PM

bking mentioned this in T342010: Test flink operations/failure scenarios relevant to Search Update Pipeline.Jul 24 2023, 3:29 PM

Gehel triaged this task as High priority.Jul 24 2023, 3:30 PM

Gehel moved this task from needs triage to Current work on the Discovery-Search board.

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Gehel updated the task description. (Show Details)Jul 24 2023, 3:43 PM

Gehel set Final Story Points to 8.

Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

Gehel edited parent tasks, added: T340548: [EPIC] Deployment of the Search Update Pipeline on Flink / k8s; removed: T317045: [Epic] Re-architect the Search Update Pipeline.Jul 25 2023, 1:38 PM

bking added a subtask: T344614: Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster.Aug 21 2023, 2:33 PM

bking updated the task description. (Show Details)Aug 28 2023, 8:54 PM

bking added a subscriber: tchin.

Gehel moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.Aug 29 2023, 1:33 PM

Gehel moved this task from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.Sep 11 2023, 3:40 PM

Some observed behavior from T344614 ,

Flink-app will start even when HA is misconfigured, which could lead the operator to believe HA was working when it actually wasn't. Is there a way to test/replicate this?

bking mentioned this in T344614: Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster.Sep 15 2023, 3:42 PM

bking updated the task description. (Show Details)Sep 15 2023, 3:45 PM

bking added a subtask: T346456: Improve concurrency limits configuration of the wdqs updater.Sep 15 2023, 5:13 PM

Change 957967 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: start adding per-env ZK path root

https://gerrit.wikimedia.org/r/957967

gerritbot added a project: Patch-For-Review.Sep 15 2023, 5:58 PM

bking updated the task description. (Show Details)Sep 18 2023, 2:55 PM

Once the patch above is merged, I think we'll need to do a little operations work:

Stop the app (helmfile destroy)
Copy the /flink znode data to our new path /dse-k8s-eqiad/rdf-streaming-updater/wdqs*
Redeploy the app

*I'm not sure how to do this. Some ZK CLI/telnet interface commands that work on earlier Zookeepers don't seem to work on our flink-zk Bookworm installation (3.8.0, as opposed to 3.4.13 on Bullseye).
I haven't had time to look closely, but I think it's related to the Four Letter Word restrictions as listed here . ~~That being said, get doesn't seem to work either.~~ Corrected, this was a matter of understanding what the get command actually does. Still unsure how to dump/restore znode data though.

bking added a subscriber: elukey.Sep 19 2023, 11:17 PM

bking added a parent task: T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.Sep 20 2023, 1:40 PM

Gehel removed a parent task: T340548: [EPIC] Deployment of the Search Update Pipeline on Flink / k8s.Sep 20 2023, 1:41 PM

Gehel closed subtask T344614: Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster as Resolved.

Gehel removed a subtask: T346456: Improve concurrency limits configuration of the wdqs updater.Sep 20 2023, 1:47 PM

Change 957967 merged by Bking:

[operations/deployment-charts@master] rdf-streaming-updater: start adding per-env ZK path root

https://gerrit.wikimedia.org/r/957967

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2023, 2:11 PM

Change 959790 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] dse-k8s: Trigger flink-app savepoint

https://gerrit.wikimedia.org/r/959790

gerritbot added a project: Patch-For-Review.Sep 21 2023, 3:29 PM

Change 959790 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s: Trigger flink-app savepoint

https://gerrit.wikimedia.org/r/959790

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2023, 5:31 PM

Change 960080 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] dse-k8s: Manually restore flink-app

https://gerrit.wikimedia.org/r/960080

gerritbot added a project: Patch-For-Review.Sep 22 2023, 3:26 PM

Change 960080 merged by Bking:

[operations/deployment-charts@master] dse-k8s: Manually restore flink-app

https://gerrit.wikimedia.org/r/960080

Maintenance_bot removed a project: Patch-For-Review.Sep 22 2023, 3:31 PM

Change 960090 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] dse-k8s: Trigger flink-app savepoint

https://gerrit.wikimedia.org/r/960090

Change 960087 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] dse-k8s: Trigger flink-app savepoint

https://gerrit.wikimedia.org/r/960087

Change 960090 abandoned by Bking:

[operations/deployment-charts@master] dse-k8s: Trigger flink-app savepoint

Reason:

patch already covered by 960087

https://gerrit.wikimedia.org/r/960090

Change 960087 merged by Bking:

[operations/deployment-charts@master] dse-k8s: Trigger flink-app savepoint

https://gerrit.wikimedia.org/r/960087

bking mentioned this in T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.Sep 22 2023, 3:55 PM

bking updated the task description. (Show Details)

Maintenance_bot removed a project: Patch-For-Review.Sep 22 2023, 4:11 PM

bking updated the task description. (Show Details)Sep 22 2023, 4:57 PM

bking updated the task description. (Show Details)Sep 22 2023, 5:00 PM

bking claimed this task.Sep 26 2023, 1:16 PM

bking updated the task description. (Show Details)Sep 26 2023, 2:29 PM

Change 963057 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: take a savepoint manually

https://gerrit.wikimedia.org/r/963057

gerritbot added a project: Patch-For-Review.Oct 3 2023, 1:37 PM

Change 963057 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: take a savepoint manually

https://gerrit.wikimedia.org/r/963057

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2023, 2:11 PM

bking added a subtask: T326914: Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink.Oct 3 2023, 5:18 PM

Moving to blocked, as we cannot start the test service until T326914 is resolved.

We're unblocked now, and we were able to test some flink operations.

Operation: Temporarily roll back breaking chart changes without rolling back in SCM.
- Result: Success.

Operation: Stop the service with a savepoint
- Result: Success, but due to recent chart changes the s3 bucket path changed. We had to manually create the bucket using swiftly head ${bucket} before the application would restart cleanly. The error manifested as a 404.

Operation: Delete flink-operator deployment.
- Result: The flink application continued to work.

Operation: Delete flink taskmanager pod while flink-operator was not running.
- Result: Kubernetes created a new taskmanager pod and the application continued to work.

Operation: Redeploy flink-operator
- Result: We had some trouble getting this to work. Destroying the deploy leaves some resources behind. To redeploy correctly, it's necessary to:
  - helmfile -e dse-k8s-eqiad -l name=flink-operator -i destroy from the admin_ng directory, followed by
  - helmfile -e dse-k8s-eqiad -l name=flink-operator -i apply

You can verify the health of the application by looking at the logs: kubectl logs -l component=jobmanager -c flink-main-container -f. If the logs show triggered and completed savepoints, the application is healthy:

{"@timestamp":"2023-10-05T14:55:45.723Z","log.level": "INFO","message":"Triggering checkpoint 35130 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1696517745715 for job 57136eb7d596a1739171fe7b360efd02.", "ecs.version": "1.2.0","process.thread.name":"Checkpoint Timer","log.logger":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator"}

{"@timestamp":"2023-10-05T14:58:19.441Z","log.level": "INFO","message":"Marking checkpoint 35130 as completed for source Source: KafkaSource:eqiad.mediawiki.page-suppress.", "ecs.version": "1.2.0","process.thread.name":"SourceCoordinator-Source: KafkaSource:eqiad.mediawiki.page-suppress","log.logger":"org.apache.flink.runtime.source.coordinator.SourceCoordinator"}

At this point, I am confident that we have enough information to properly manage the service, and I believe we are ready to start deploying in staging. As such, I'm resolving this ticket. However, if @dcausse , @pfischer , @gmodena or anyone else feel like more testing is needed, please feel free to reopen.

bking updated the task description. (Show Details)Oct 5 2023, 3:13 PM

@bking thanks for taking care of this! Something I can't remember if this was done or not is simulating a k8s upgrade and see if the job recovers from its latest checkpoint (hopefully still available in zk).
The procedure could look like:

have the job running
undeploy
inspect the namespace and make sure that it's empty
redeploy the job and test if it picks up the latest checkpoint from zookeeper

In T342149#9230691, @dcausse wrote:

@bking thanks for taking care of this! Something I can't remember if this was done or not is simulating a k8s upgrade and see if the job recovers from its latest checkpoint (hopefully still available in zk).
The procedure could look like:

have the job running

undeploy

inspect the namespace and make sure that it's empty

redeploy the job and test if it picks up the latest checkpoint from zookeeper

Tested this procedure above and it did not work.
Undeploying via helmfile probably told the flink-k8s-operator to cancel the job and it might have discarded the H/A metadata because when re-deploying it attempted to start from the initialSavePointPath provided in that values.yaml file instead of the latest known checkpoint.
Recovering the job required going thru the Manual recovery procedure (identifying the checkpoint manually) and deploying with:
helmfile -e dse-k8s-eqiad --selector name=wikidata -i apply --set app.job.initialSavepointPath=s3://rdf-streaming-updater-staging-k8s-op-test-dse/wikidata/checkpoints/57136eb7d596a1739171fe7b360efd02/chk-37424
So I think we still need to figure out what is the procedure to apply during k8s cluster upgrades.

I think the semantic of destroying the flinkdeployment resource is to get rid of the job and this is what the flink k8s operator will do.
So the k8s upgrade procedure must not undeploy these (at least not while the k8s operator is running).
We might ask service ops what happens during k8s upgrades and how they bring down the cluster, if during that step they undeploy individual services (via helmfile destroy) this might not work for flink-app. If not we should try to simulate what they'll be doing to see if the native recovery mechanism will survive a re-deploy.

@JMeybohm Is it possible for us to simulate a cluster upgrade (maybe by running the upgrade cookbook against a testing cluster?) Based on what @elukey mentioned in #wikimedia-k8s-sig IRC, it sounds like we'll be OK. But it would definitely be better to have more certainty before we're in production.

It should be possible to do the cluster upgrade procedure without an actual update, yes ("simulation" does not seem appropriate though as you will still delete all of etcd). But I still wonder if what @dcausse describes is desired. Maybe the operator can be instructed to never "purge" data? Seems a bit more safe to me.

In T342149#9234617, @JMeybohm wrote:

It should be possible to do the cluster upgrade procedure without an actual update, yes ("simulation" does not seem appropriate though as you will still delete all of etcd). But I still wonder if what @dcausse describes is desired. Maybe the operator can be instructed to never "purge" data? Seems a bit more safe to me.

The operator will purge the H/A metadata if the flindeployment is explicitly removed, I think this makes sense from the flink-k8s-operator perspective. It does not appear to have an option to tell it to keep the H/A metadata even-though we tell it to delete the deployment, the sole option it has is to take a very last savepoint before the cleanup.

@JMeybohm are all services (after beeing depooled) explicitly "undeployed" (via helmfile destroy) as part of the procedure to bring the upgraded k8s cluster down or are they going to simply die because of lack worker nodes available?

In T342149#9234816, @dcausse wrote:

The operator will purge the H/A metadata if the flindeployment is explicitly removed, I think this makes sense from the flink-k8s-operator perspective. It does not appear to have an option to tell it to keep the H/A metadata even-though we tell it to delete the deployment, the sole option it has is to take a very last savepoint before the cleanup.

Makes sense indeed.

In T342149#9234816, @dcausse wrote:

@JMeybohm are all services (after beeing depooled) explicitly "undeployed" (via helmfile destroy) as part of the procedure to bring the upgraded k8s cluster down or are they going to simply die because of lack worker nodes available?

No actions will be taken to undeploy anything at all. After depooling the workload the etcd contents will be wiped and the workload will just die at some point (when the it's running on is re-imaged at latest).

In T342149#9234852, @JMeybohm wrote:

In T342149#9234816, @dcausse wrote:

@JMeybohm are all services (after beeing depooled) explicitly "undeployed" (via helmfile destroy) as part of the procedure to bring the upgraded k8s cluster down or are they going to simply die because of lack worker nodes available?

No actions will be taken to undeploy anything at all. After depooling the workload the etcd contents will be wiped and the workload will just die at some point (when the it's running on is re-imaged at latest).

Thanks! so I believe we should be good, the H/A metadata can't go away if the operator is not explicitly told to do so. It is my test scenario (using helmfile destroy) that was not correct.

To be certain that it'll work I guess we could manually bring this workload down (the operator and the rdf-streaming-updater job) and then manually cleanup the rdf-streaming-updater namespace while the operator is down.

bking reopened this task as In Progress.Oct 12 2023, 10:35 PM

bking moved this task from Done to In Progress on the Data-Platform-SRE board.

Gehel closed subtask T326914: Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink as Resolved.Oct 13 2023, 12:58 PM

Gehel moved this task from Ready for Dev -- SRE/Ops to Blocked/Waiting on the Discovery-Search (Current work) board.Oct 20 2023, 8:21 AM

bking moved this task from Blocked/Waiting to In Progress on the Discovery-Search (Current work) board.Oct 23 2023, 3:06 PM

We might still want to test a k8s upgrade, but this should not be blocking moving to production. Worst case, we can do a manual recovery.

Gehel moved this task from In Progress to Blocked / Waiting on the Data-Platform-SRE board.Oct 31 2023, 4:34 PM