Page MenuHomePhabricator
Create Task
Maniphest T412978

Support for Java 21 and Flink 2
Closed, ResolvedPublic
Actions

Description

Related to migrating Pyflink to Java and a new HTML enrichment pipeline, we are starting with a new setup of Java & Flink and we would like to take the opportunity to do it in the latest versions, rather than needing to migrate in the future.

Right now, Java 17 and Flink 1.2 are supported, but we don't have the tools to develop in Java 21 and Flink 2.

We need to at least prepare the following:

Task is done if we can deploy Flink pipelines on Flink 2 with Java 21.

Details

Other Assignee
brouberol
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
images: maven-21JavierMontonintegration/configmaster+48 -0
flink: updating control to jdk21Atsukooperations/docker-images/production-imagesmaster+1 -1
eventutilitites-flink: Remove deprecated classesJavierMontonwikimedia-event-utilitiesmaster+1 -146
image: Flink 2JavierMontonoperations/docker-images/production-imagesmaster+8 -1
stream: webrequest-page-view-nextJavierMontonoperations/deployment-chartsmaster+1 -1
image: Flink 2JavierMontonoperations/docker-images/production-imagesmaster+9 -3
jvm: eventutilitites-flinkJavierMontonwikimedia-event-utilitiesmaster+15 -16
flink: build the flink 2.0.0 imageBrouberoloperations/docker-images/production-imagesmaster+7 -3
dockerfiles: Add maven-java25JavierMontonintegration/configmaster+34 -0
dockerfiles: Add java25 imageJavierMontonintegration/configmaster+15 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Upgrade dependenciesrepos/maven/wmf-jvm-parent-pom!29javiermontonfeature/upgrade-dependenciesmain
New profile for Java 25.repos/maven/wmf-jvm-parent-pom!28javiermontonfeature/upgrade-parent-pom-to-jdk-25main
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT426863 Flink 2 migration
 ResolvedJMonton-WMFT412978 Support for Java 21 and Flink 2
 ResolvedatsukoT427774 Flink image fails to build in weekly rebuild

Event Timeline

JMonton-WMF created this task.Dec 17 2025, 4:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 17 2025, 4:10 PM
JMonton-WMF edited projects, added: Data-Engineering (Q2 FY25/26 October 1st - December 31th); removed: Data-Engineering.Dec 17 2025, 4:10 PM
JMonton-WMF edited projects, added: Data-Engineering; removed: Data-Engineering (Q2 FY25/26 October 1st - December 31th), Java-Scala-Standardization.
JMonton-WMF edited projects, added: Data-Engineering (Q2 FY25/26 October 1st - December 31th); removed: Data-Engineering.
JMonton-WMF moved this task from Next Up to In progress on the Data-Engineering (Q2 FY25/26 October 1st - December 31th) board.
gerritbot added a comment.Dec 17 2025, 4:37 PM

Change #1219178 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[integration/config@master] component: dockerfiles

https://gerrit.wikimedia.org/r/1219178

gerritbot added a project: Patch-For-Review.Dec 17 2025, 4:37 PM
gerritbot added a comment.Dec 17 2025, 5:09 PM

Change #1219184 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[integration/config@master] component: dockerfiles

https://gerrit.wikimedia.org/r/1219184

CodeReviewBot added a comment.Dec 17 2025, 5:15 PM

javiermonton updated https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/28

New profile for Java 25.

CodeReviewBot added a comment.Dec 17 2025, 5:16 PM

javiermonton updated https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/29

Upgrade dependencies

JMonton-WMF updated the task description. (Show Details)Dec 18 2025, 11:22 AM
CodeReviewBot added a comment.Dec 18 2025, 11:49 AM

javiermonton merged https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/29

Upgrade dependencies

gerritbot added a comment.Dec 18 2025, 4:04 PM

Change #1219178 merged by jenkins-bot:

[integration/config@master] dockerfiles: Add java25 image

https://gerrit.wikimedia.org/r/1219178

gerritbot added a comment.Dec 18 2025, 4:23 PM

Change #1219184 merged by jenkins-bot:

[integration/config@master] dockerfiles: Add maven-java25

https://gerrit.wikimedia.org/r/1219184

JMonton-WMF updated the task description. (Show Details)Dec 19 2025, 9:13 AM
gmodena subscribed.Dec 19 2025, 9:53 AM
Ahoelzl edited projects, added: Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed: Data-Engineering (Q2 FY25/26 October 1st - December 31th).Jan 8 2026, 4:31 PM
Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.
JMonton-WMF moved this task from In progress to Blocked/Paused on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Jan 23 2026, 7:20 AM
CodeReviewBot added a comment.Jan 23 2026, 2:41 PM

javiermonton merged https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/28

New profile for Java 25.

JMonton-WMF updated the task description. (Show Details)Jan 23 2026, 2:42 PM
JAllemandou moved this task from Blocked/Paused to Next Up on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Jan 26 2026, 3:28 PM
Ahoelzl edited projects, added: Data-Engineering (Q4 FS25/26 April 1st - June 30st); removed: Data-Engineering (Q3 FY25/26 January 1st - March 31th).Apr 3 2026, 10:23 PM
brouberol updated Other Assignee, added: brouberol.May 11 2026, 7:29 AM
gerritbot added a comment.May 11 2026, 7:35 AM

Change #1285540 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/docker-images/production-images@master] flink: build the flink 2.0.0 image

https://gerrit.wikimedia.org/r/1285540

brouberol changed the task status from Open to In Progress.May 11 2026, 7:37 AM
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
brouberol added a project: Data-Platform-SRE (2026-04-24 - 2026-05-15).
gerritbot added a comment.May 11 2026, 9:06 AM

Change #1285540 merged by Brouberol:

[operations/docker-images/production-images@master] flink: build the flink 2.0.0 image

https://gerrit.wikimedia.org/r/1285540

Ottomata subscribed.May 11 2026, 10:24 AM

Change #1285540 merged by Brouberol [...] flink: build the flink 2.0.0 image

😍

gerritbot added a comment.May 11 2026, 3:07 PM

Change #1285845 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[wikimedia-event-utilities@master] jvm: eventutilitites-flink

https://gerrit.wikimedia.org/r/1285845

JMonton-WMF moved this task from Next Up to In progress on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.May 11 2026, 3:41 PM
gerritbot added a comment.May 12 2026, 2:21 PM

Change #1285845 merged by jenkins-bot:

[wikimedia-event-utilities@master] jvm: eventutilitites-flink

https://gerrit.wikimedia.org/r/1285845

gerritbot added a comment.May 19 2026, 9:06 AM

Change #1289278 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/docker-images/production-images@master] image: Flink 2

https://gerrit.wikimedia.org/r/1289278

gerritbot added a comment.May 19 2026, 9:23 AM

Change #1289278 merged by Brouberol:

[operations/docker-images/production-images@master] image: Flink 2

https://gerrit.wikimedia.org/r/1289278

JMonton-WMF added a comment.May 20 2026, 1:08 PM

So far everything is working with Flink 2.0.2 and Java 25, docker images are built and tests are passing, but we haven't tested a real deployment in K8s yet.

This may be a big concern: https://issues.apache.org/jira/browse/FLINK-37719

JMonton-WMF updated the task description. (Show Details)May 20 2026, 1:46 PM
JMonton-WMF added a parent task: T426863: Flink 2 migration.May 20 2026, 1:57 PM
gerritbot added a comment.May 21 2026, 3:44 PM

Change #1290822 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] stream: webrequest-page-view-next

https://gerrit.wikimedia.org/r/1290822

gerritbot added a comment.May 21 2026, 3:56 PM

Change #1290822 merged by jenkins-bot:

[operations/deployment-charts@master] stream: webrequest-page-view-next

https://gerrit.wikimedia.org/r/1290822

gerritbot added a comment.May 26 2026, 9:28 AM

Change #1293664 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/docker-images/production-images@master] image: Flink 2

https://gerrit.wikimedia.org/r/1293664

atsuko subscribed.May 26 2026, 3:43 PM

Notes about apache-flink==2.0.2 bot working on python3.13 on trixie.

gerritbot added a comment.May 26 2026, 3:53 PM

Change #1293758 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[integration/config@master] images: maven-21

https://gerrit.wikimedia.org/r/1293758

JMonton-WMF added a comment.May 26 2026, 4:04 PM

We've exploring how to build an image with Flink 2.0.2 and Java 25 and we've been having some issues. The Java 25 base image is based on Trixie, it needs a few changes in our docker template, but the Python setup is broken and fails for PyFlink.
We've considered backporting Java 25 to bookworm, but it isn't currently available on our packages. We've considered copying Python 3.11 from Bookworm to Trixie too, but it needs a lot of changes, and still, we may find future issues with Java 25 and Flink.

We are moving now to use Java 21 with Flink 2.0.2. The docker image we can build it using Bookworm as we already have the package available, and it won't break PyFlink.

gerritbot added a comment.May 27 2026, 1:11 PM

Change #1293664 merged by Atsuko:

[operations/docker-images/production-images@master] image: Flink 2

https://gerrit.wikimedia.org/r/1293664

gerritbot added a comment.May 28 2026, 1:23 PM

Change #1294993 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[wikimedia-event-utilities@master] eventutilitites-flink: Remove deprecated classes

https://gerrit.wikimedia.org/r/1294993

JMonton-WMF renamed this task from Support for Java 25 and Flink 2 to Support for Java 21 and Flink 2.May 28 2026, 1:47 PM
JMonton-WMF updated the task description. (Show Details)
gerritbot added a comment.May 28 2026, 1:48 PM

Change #1294993 merged by jenkins-bot:

[wikimedia-event-utilities@master] eventutilitites-flink: Remove deprecated classes

https://gerrit.wikimedia.org/r/1294993

JMonton-WMF updated the task description. (Show Details)May 28 2026, 1:48 PM
atsuko added a subtask: T427774: Flink image fails to build in weekly rebuild.Jun 1 2026, 9:22 AM
gerritbot added a comment.Jun 1 2026, 9:29 AM

Change #1295860 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/docker-images/production-images@master] flink: updating control to jdk21

https://gerrit.wikimedia.org/r/1295860

gerritbot added a comment.Jun 1 2026, 9:34 AM

Change #1295860 merged by Atsuko:

[operations/docker-images/production-images@master] flink: updating control to jdk21

https://gerrit.wikimedia.org/r/1295860

atsuko closed subtask T427774: Flink image fails to build in weekly rebuild as Resolved.Jun 1 2026, 10:53 AM
JMonton-WMF moved this task from In progress to In Review on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.Jun 1 2026, 3:14 PM
JMonton-WMF updated the task description. (Show Details)Jun 2 2026, 9:43 AM
JMonton-WMF moved this task from In Review to Done on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.
gerritbot added a comment.Jun 4 2026, 8:31 AM

Change #1293758 merged by jenkins-bot:

[integration/config@master] images: maven-21

https://gerrit.wikimedia.org/r/1293758

Stashbot added a comment.Jun 4 2026, 8:35 AM

Mentioned in SAL (#wikimedia-releng) [2026-06-04T08:35:41Z] <hashar> Built Docker images docker-registry.wikimedia.org/releng/java21:0.1 and docker-registry.wikimedia.org/releng/maven-java21:0.1 # T412978

hashar subscribed.Jun 4 2026, 9:30 AM

I have built the images:

docker-registry.wikimedia.org/releng/java21:0.1
docker-registry.wikimedia.org/releng/maven-java21:0.1

Which should now be usable in GitLab pipelines. Thank you very much @JMonton-WMF for the very detailed context you gave me on chat!

Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE (2026-04-24 - 2026-05-15).Jun 5 2026, 1:27 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Ahoelzl closed this task as Resolved.Jun 30 2026, 4:27 PM
Gehel moved this task from In Progress to Done on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.Jul 3 2026, 7:23 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.Jul 3 2026, 7:24 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits