Page MenuHomePhabricator
Create Task
Maniphest T336185

Enable HA failover for flink-kubernetes-operator
Closed, ResolvedPublic
Actions

Description

flink-kubernetes-operator supports running multiple replicas in HA standby mode.

https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/configuration/#leader-election-and-high-availability

We should enable this in DSE, and run with two or 3 replicas. Once we do this for a little while, we should do the same as we deploy in wikikube in T333464.

Running without HA in normal operation is fine
https://phabricator.wikimedia.org/T324576#8454404

But we should enable HA standby replicas for good measure anyway.

Details

SubjectRepoBranchLines +/-
Fix overlapping names edge case in flink-operatoroperations/deployment-chartsmaster+3 -8
flink-operator - enable HA leader election and set replicas: 2operations/deployment-chartsmaster+9 -10
Allow managing leases in flink-operator namespace when using HAoperations/deployment-chartsmaster+43 -0
flink-operator - disable HA replicas for nowoperations/deployment-chartsmaster+10 -8
flink-operator - enable HA leader election and set replicas: 2operations/deployment-chartsmaster+19 -0
Customize query in gerrit

Related Objects

Event Timeline

Ottomata created this task.May 8 2023, 5:00 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptMay 8 2023, 5:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ottomata added a subscriber: JMeybohm.May 8 2023, 5:00 PM
gerritbot added a comment.May 8 2023, 5:56 PM

Change 917389 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] flink-operator - enable HA leader election and set replicas: 2

https://gerrit.wikimedia.org/r/917389

gerritbot added a project: Patch-For-Review.May 8 2023, 5:56 PM
gerritbot added a comment.May 8 2023, 6:05 PM

Change 917389 merged by jenkins-bot:

[operations/deployment-charts@master] flink-operator - enable HA leader election and set replicas: 2

https://gerrit.wikimedia.org/r/917389

Maintenance_bot removed a project: Patch-For-Review.May 8 2023, 6:10 PM
Ottomata added a comment.May 8 2023, 6:11 PM

Hm, looks like i'm going to need to add (or enable?) some perms to manage leases in the flink-operator namespace:

User \"system:serviceaccount:flink-operator:flink-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"flink-operator\"
Ottomata moved this task from Backlog to Sprint 12 on the Event-Platform board.May 8 2023, 7:12 PM
Ottomata edited projects, added Event-Platform (Sprint 12); removed Event-Platform.
Ottomata added a comment.May 8 2023, 8:30 PM

Asking flink user mailing list.
https://lists.apache.org/thread/yq89jm0szkcodfocm5x7vqnqdmh0h1l0

Ottomata added a comment.May 9 2023, 2:29 PM

Its a bug.

https://issues.apache.org/jira/browse/FLINK-32041

gerritbot added a comment.May 9 2023, 3:26 PM

Change 917907 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] flink-operator - disable HA replicas for now

https://gerrit.wikimedia.org/r/917907

gerritbot added a project: Patch-For-Review.May 9 2023, 3:26 PM
gerritbot added a comment.May 9 2023, 3:34 PM

Change 917907 merged by jenkins-bot:

[operations/deployment-charts@master] flink-operator - disable HA replicas for now

https://gerrit.wikimedia.org/r/917907

Maintenance_bot removed a project: Patch-For-Review.May 9 2023, 4:11 PM
Ottomata moved this task from Next Up to In progress on the Event-Platform (Sprint 12) board.May 9 2023, 9:34 PM
Ottomata moved this task from In progress to Next Up on the Event-Platform (Sprint 12) board.
JArguello-WMF edited projects, added Event-Platform (Sprint 14 A); removed Event-Platform (Sprint 12).May 12 2023, 3:52 PM
JArguello-WMF triaged this task as Low priority.May 15 2023, 3:17 PM
gerritbot added a comment.May 18 2023, 4:31 PM

Change 921064 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] Allow managing leases in flink-operator namespace when using HA

https://gerrit.wikimedia.org/r/921064

gerritbot added a project: Patch-For-Review.May 18 2023, 4:31 PM
gerritbot added a comment.May 18 2023, 5:09 PM

Change 921064 merged by jenkins-bot:

[operations/deployment-charts@master] Allow managing leases in flink-operator namespace when using HA

https://gerrit.wikimedia.org/r/921064

Maintenance_bot removed a project: Patch-For-Review.May 18 2023, 5:10 PM
gerritbot added a comment.May 18 2023, 5:13 PM

Change 921067 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] flink-operator - enable HA leader election and set replicas: 2

https://gerrit.wikimedia.org/r/921067

gerritbot added a project: Patch-For-Review.May 18 2023, 5:13 PM
gerritbot added a comment.May 18 2023, 5:28 PM

Change 921067 merged by Ottomata:

[operations/deployment-charts@master] flink-operator - enable HA leader election and set replicas: 2

https://gerrit.wikimedia.org/r/921067

Maintenance_bot removed a project: Patch-For-Review.May 18 2023, 5:30 PM
Ottomata assigned this task to tchin.May 22 2023, 1:25 PM
Ottomata moved this task from Next Up to In progress on the Event-Platform (Sprint 14 A) board.
JArguello-WMF moved this task from In progress to In Review on the Event-Platform (Sprint 14 A) board.May 24 2023, 1:21 PM
gerritbot added a comment.May 31 2023, 8:31 PM

Change 925016 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] Fix overlapping names edge case in flink-operator

https://gerrit.wikimedia.org/r/925016

gerritbot added a project: Patch-For-Review.May 31 2023, 8:31 PM
JArguello-WMF edited projects, added Event-Platform (Sprint 14 B); removed Event-Platform (Sprint 14 A).Jun 5 2023, 11:42 AM
JArguello-WMF moved this task from Next Up to In Review on the Event-Platform (Sprint 14 B) board.
JArguello-WMF moved this task from In Review to Done on the Event-Platform (Sprint 14 B) board.Jun 5 2023, 3:06 PM
gerritbot added a comment.Jun 5 2023, 7:05 PM

Change 925016 abandoned by TChin:

[operations/deployment-charts@master] Fix overlapping names edge case in flink-operator

Reason:

Abandoned in favor of waiting for upstream fix to be released

https://gerrit.wikimedia.org/r/925016

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2023, 7:11 PM
JArguello-WMF closed this task as Resolved.Jun 20 2023, 3:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits