Page MenuHomePhabricator
Create Task
Maniphest T403838

Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11
Closed, ResolvedPublic
Actions

Description

We are running the 1.12.0 version of the flink-k8s-operator and it appears that this version is hit by a critical bug that will prevent jobs from upgrading (deploying new version): https://issues.apache.org/jira/browse/FLINK-38033

The bug appears fixed in 1.12.1 so we could upgrade to this or possibly downgrade to 1.11.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
flink: build flink 1.20 on top of bookworm/jdk 17Brouberoloperations/docker-images/production-imagesmaster+11 -5
flink-kubernetes-operator: upgrade to 1.12.1 (fix typo)Brouberoloperations/deployment-chartsmaster+1 -1
flink-kubernetes-operator: upgrade to 1.12.1Brouberoloperations/deployment-chartsmaster+1 -16
flink-kubernetes-operator: upgrade to 1.12.1 in dse-k8s-eqiadBrouberoloperations/deployment-chartsmaster+3 -0
flink-kubernetes-operator: upgrade to 1.12.1 in codfwBrouberoloperations/deployment-chartsmaster+3 -0
flink-kubernetes-operator: upgrade to 1.12.1 in eqiadBrouberoloperations/deployment-chartsmaster+3 -0
flink-kubernetes-operator: upgrade to 1.12.1 in staging-codfwBrouberoloperations/deployment-chartsmaster+3 -0
flink-kubernetes-operator: upgrade to 1.12.1 in staging-eqiadBrouberoloperations/deployment-chartsmaster+3 -0
flink-kubernetes-operator: upgrade to 1.12.1Brouberoloperations/docker-images/production-imagesmaster+7 -1
Customize query in gerrit

Related Objects

Event Timeline

dcausse created this task.Sep 5 2025, 3:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 5 2025, 3:28 PM
dcausse renamed this task from Upgrade flink-k8s-operator to 1.12.1 or downgrade 1.11 to Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11.Sep 5 2025, 3:29 PM
dcausse triaged this task as High priority.
bking subscribed.Sep 5 2025, 3:29 PM
Gehel edited projects, added: Data-Platform-SRE (2025.09.05 - 2025.09.26), Data-Engineering; removed: Data-Platform-SRE.Sep 5 2025, 3:32 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
dcausse updated the task description. (Show Details)Sep 5 2025, 3:33 PM
dcausse mentioned this in T372912: Migrate image recommendation to use page_weighted_tags_changed stream.Sep 5 2025, 3:43 PM
dcausse updated the task description. (Show Details)Sep 8 2025, 6:58 AM
gerritbot added a comment.Sep 8 2025, 7:54 AM

Change #1185846 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/docker-images/production-images@master] flink-kubernetes-operator: upgrade to 1.12.1

https://gerrit.wikimedia.org/r/1185846

gerritbot added a project: Patch-For-Review.Sep 8 2025, 7:54 AM
gerritbot added a comment.Sep 8 2025, 8:04 AM

Change #1185846 merged by Brouberol:

[operations/docker-images/production-images@master] flink-kubernetes-operator: upgrade to 1.12.1

https://gerrit.wikimedia.org/r/1185846

brouberol subscribed.Sep 8 2025, 8:07 AM
~ ❯ docker pull docker-registry.discovery.wmnet/flink-kubernetes-operator:1.12.1-wmf0
1.12.1-wmf0: Pulling from flink-kubernetes-operator
653eb674aee2: Pull complete
ba36c88ab901: Downloading [========================>                          ]  37.14MB/74.71MB

The docker-registry.discovery.wmnet/flink-kubernetes-operator:1.12.1-wmf0 image was published and is now available.

gerritbot added a comment.Sep 8 2025, 8:12 AM

Change #1185849 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in staging-codfw

https://gerrit.wikimedia.org/r/1185849

gerritbot added a comment.Sep 8 2025, 8:12 AM

Change #1185850 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in staging-eqiad

https://gerrit.wikimedia.org/r/1185850

gerritbot added a comment.Sep 8 2025, 8:12 AM

Change #1185851 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in codfw

https://gerrit.wikimedia.org/r/1185851

gerritbot added a comment.Sep 8 2025, 8:12 AM

Change #1185852 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in eqiad

https://gerrit.wikimedia.org/r/1185852

gerritbot added a comment.Sep 8 2025, 8:12 AM

Change #1185853 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1185853

gerritbot added a comment.Sep 8 2025, 8:12 AM

Change #1185854 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1

https://gerrit.wikimedia.org/r/1185854

brouberol claimed this task.Sep 8 2025, 8:14 AM
gerritbot added a comment.Sep 8 2025, 8:20 AM

Change #1185850 merged by Brouberol:

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in staging-eqiad

https://gerrit.wikimedia.org/r/1185850

brouberol changed the task status from Open to In Progress.Sep 8 2025, 8:52 AM
brouberol moved this task from Backlog - operations to Needs Review on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
dcausse added a comment.EditedSep 8 2025, 9:05 AM

Done some testing :
reproduce the bug on cirrus-streaming-updater (done twice)

  1. destroy the job: helmfile destroy -e staging -i => OK
  2. deploy the job: helmfile deploy -e staging -i => OK
  3. restart the job: helmfile apply -e staging -i --selector name=producer --set app.restartNonce=2 => OK the job recovered

test suspend/resume

  1. suspend the job: helmfile apply -e staging -i --selector name=producer --set app.job.state=suspended => OK the job stopped, small caveat the jobmanager pod remained alive
  2. resume the job: helmfile apply -e staging -i --selector name=producer --set app.job.state=running => OK, the stale jobmanager pod disappeared and new one was created

test manual savepoint

  • take a savepoint: helmfile apply -e staging -i --selector name=producer --set app.job.savepointTriggerNonce=2 => OK, savepoint taken

test restarting another job without destroying the release first (rdf-streaming-updater)

  • restart the job: helmfile apply -e staging -i --selector name=wikidata --set app.restartNonce=2 => OK the job restarted on the right savepoint

@brouberol thanks, the bug is gone and all normal operations appear to work OK in staging.
A minor annoyance is that when we suspend the job flink does not appear to be willing to finish the jobmanager pod, it might stay idle. We rarely suspend for a long time so I believe that this is acceptable. I haven't done that in a while so it's very likely that it is an existing problem and probably not directly related to the operator but more to flink not willing to stop when it goes idle.

gerritbot added a comment.Sep 8 2025, 9:33 AM

Change #1185849 merged by jenkins-bot:

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in staging-codfw

https://gerrit.wikimedia.org/r/1185849

gerritbot added a comment.Sep 8 2025, 9:33 AM

Change #1185852 merged by jenkins-bot:

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in eqiad

https://gerrit.wikimedia.org/r/1185852

gerritbot added a comment.Sep 8 2025, 9:33 AM

Change #1185851 merged by jenkins-bot:

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in codfw

https://gerrit.wikimedia.org/r/1185851

gerritbot added a comment.Sep 8 2025, 9:33 AM

Change #1185853 merged by jenkins-bot:

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 in dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1185853

gerritbot added a comment.Sep 8 2025, 9:34 AM

Change #1185854 merged by jenkins-bot:

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1

https://gerrit.wikimedia.org/r/1185854

gerritbot added a comment.Sep 8 2025, 9:55 AM

Change #1185872 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 (fix typo)

https://gerrit.wikimedia.org/r/1185872

gerritbot added a comment.Sep 8 2025, 10:49 AM

Change #1185872 merged by Brouberol:

[operations/deployment-charts@master] flink-kubernetes-operator: upgrade to 1.12.1 (fix typo)

https://gerrit.wikimedia.org/r/1185872

brouberol added a comment.Sep 8 2025, 11:08 AM

All operators have been upgraded.

brouberol moved this task from Needs Review to Blocked/Waiting on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.Sep 8 2025, 11:09 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 8 2025, 11:31 AM
dcausse added a comment.Sep 8 2025, 12:16 PM

Tested one job in codfw and it worked well, thanks!

brouberol closed this task as Resolved.Sep 8 2025, 2:03 PM
brouberol moved this task from Blocked/Waiting to Done on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
Gehel added a project: Essential-Work.Sep 11 2025, 12:11 PM
gerritbot added a comment.Sep 11 2025, 12:17 PM

Change #1187416 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/docker-images/production-images@master] flink: build flink 1.20 on top of bookworm/jdk 17

https://gerrit.wikimedia.org/r/1187416

gerritbot added a project: Patch-For-Review.Sep 11 2025, 12:17 PM
gerritbot added a comment.Sep 11 2025, 2:50 PM

Change #1187416 merged by Brouberol:

[operations/docker-images/production-images@master] flink: build flink 1.20 on top of bookworm/jdk 17

https://gerrit.wikimedia.org/r/1187416

Maintenance_bot removed a project: Patch-For-Review.Sep 11 2025, 3:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.Sep 12 2025, 2:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits