Currently the enrichment jobs run on Flink 1.20.1 and Java 11. This is the last docker image available that has Java 11. An upgrade to Flink 1.20.3 requires Java 17.
Simply bumping the image isn't enough since the distro in the latest images trigger PEP 668 - Marking Python base environments as "externally managed". This breaks our build step. (I think?) the way to fix this is to update Blubber and use its new use-system-site-packages option, but that is blocked because of T406872
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Bump Flink to 1.20.3 | repos/data-engineering/mediawiki-event-enrichment!121 | javiermonton | feature/bump-eventutilitiles-python | main | |
| Bump Flink to 1.20.3 | repos/data-engineering/eventutilities-python!97 | javiermonton | feature/bump-flink-to-1.20.3 | main | |
| Fix: java.net.MalformedURLException: no protocol | repos/data-engineering/eventutilities-python!96 | javiermonton | fix/malformed-url-exception | main | |
| Bump eventutilities-python to 0.21.0 | repos/data-engineering/mediawiki-event-enrichment!117 | javiermonton | feature/bump-eventutilities-python | main | |
| Upgrade to flink 1.20.2 image which uses Java 17. | repos/data-engineering/mediawiki-event-enrichment!112 | otto | jdk-17 | main | |
| Use Java 17 | repos/data-engineering/eventutilities-python!93 | tchin | use-jdk-17 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T404340 [EPIC] Upgrade flink jobs to java 17 | |||
| Resolved | JMonton-WMF | T408918 Upgrade mediawiki-event-enrichment jobs to >= Flink 1.20.3 and Java 17 | |||
| Resolved | JMonton-WMF | T406872 Fix mediawiki event enrichment to work with newest version of Blubber |
tchin opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/100
Bump Bllubber to v1.5.0, Flink to 1.20.2, and Java to 17
Okay, I've submitted another MR!94 that I think addresses a lot of the in our face CI problems.
The gist of the MR so far:
And now! Finally we are hitting some PyFlink + Java 17 (?) related issues! Everything so far has been other python dependency or CI env problems.
This test job has failed test output. I can reproduce this in local Docker as well.
It would be nice if we could have separated the CI problems from the Java upgrade, but we can't build on top of our Java 11 images anymore because they are based on debian buster, which doesn't work any more because buster-backports debian repo is gone.
So, we have to solve Java 17 issues now.
Next up: dive into the test failures to figure out what is going on.
otto updated https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/94
Refactor CI and use openjdk-17 and python 3.11 for tests and CI.
Next up: dive into the test failures to figure out what is going on.
It was apache-beam using pkg_resources after all. Pinning setuptools < 72 did it! Yeehaw!
otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/94
Refactor CI and use openjdk-17 and python 3.11 for tests and CI.
otto opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/112
Draft: Upgrade to flink 1.20.2 image which uses Java 17.
otto merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/112
Upgrade to flink 1.20.2 image which uses Java 17.
There was another issue with the new eventutilities-python version, where the argparse library was checking if a folder was writable or not, and it fails on K8s because /srv/app is not writable. It was addressed in this MR and released in version 0.20.0.
Still, after deploying 0.20.0, there is another issue, it seems to be similar to this Flink issue, but according to the docs, it was fixed.
This is the log:
java.net.MalformedURLException: no protocol: ['file:/usr/local/lib/python3.11/dist-packages/pyflink/opt/flink-python-1.20.2.jar', 'file:/usr/local/lib/python3.11/dist-packages/pyflink/opt/flink-python-1.20.2.jar', 'file:///opt/lib/venv/lib/python3.11/site-packages/eventutilities_python/lib/flink-connector-kafka-3.3.0-1.20.jar', 'file:///opt/lib/venv/lib/python3.11/site-packages/eventutilities_python/lib/eventutilities-flink-1.4.0-jar-with-dependencies.jar', 'file:///opt/lib/venv/lib/python3.11/site-packages/eventutilities_python/lib/flink-python-1.20.0-tests.jar', 'file:///opt/lib/venv/lib/python3.11/site-packages/eventutilities_python/lib/kafka-clients-3.4.0.jar']
javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/96
Fix: java.net.MalformedURLException: no protocol
This MR: https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/96 addresses the latest issue. I have published a dev0 version of this change and tested in K8s, and now it is working fine.
We still need to merge this MR, create a proper release, and include it on the applications using the failing version.
javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/117
Bump eventutilities-python to 0.21.0
Change #1248808 had a related patch set uploaded (by JavierMonton; author: JavierMonton):
[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich-next
Related to that previous MR, I believe that the issue was solved in Flink 1.20.3 with this commit. Maybe we could ensure we use that one, rather than 1.20.2.
javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/96
Fix: java.net.MalformedURLException: no protocol
javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/117
Bump eventutilities-python to 0.21.0
Change #1248808 merged by jenkins-bot:
[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich-next
@JMonton-WMF in eventutilities-python MR96 you wrote:
It seems to be similar or related to https://issues.apache.org/jira/browse/FLINK-36457, which I'm not reading it wrong, it was fixed on version 1.20.3
Should we just upgrade to 1.20.3? If available in PyPI, it should be pretty easy to do. I can make it available today probably.
Change #1249312 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/docker-images/production-images@master] flink - bump to 1.20.3 to pick up fix for FLINK-36457
Sounds good. I was a bit blocked as the previous version was failing and ended up doing that work around. But definitely upgrading to 1.20.3 looks better.
Change #1249312 merged by Btullis:
[operations/docker-images/production-images@master] flink - bump to 1.20.3 to pick up fix for FLINK-36457
@JMonton-WMF Flink image version 1.20.3-wmf0-20260309 is available! https://docker-registry.wikimedia.org/flink/tags/
javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/97
Bump Flink to 1.20.3
javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/121
Bump Flink to 1.20.3
javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/97
Bump Flink to 1.20.3
javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/121
Bump Flink to 1.20.3
Change #1251456 had a related patch set uploaded (by JavierMonton; author: JavierMonton):
[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich
Change #1251456 merged by jenkins-bot:
[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich
The workaround added in event utilities has been reverted, the version 0.22 doesn't use it anymore.
Mediawiki-event-enrichment is now using the new Flink 1.20.3 docker image. It's deployed with the HTML enrichment pipeline, but not on the others yet.
I'll deploy the others on Monday.
Change #1253425 had a related patch set uploaded (by JavierMonton; author: JavierMonton):
[operations/deployment-charts@master] stream: mw-content-history-reconcile-enrich
Change #1253426 had a related patch set uploaded (by JavierMonton; author: JavierMonton):
[operations/deployment-charts@master] stream: mw-page-content-change-enrich
Change #1253425 merged by jenkins-bot:
[operations/deployment-charts@master] stream: mw-content-history-reconcile-enrich
Change #1253426 merged by jenkins-bot:
[operations/deployment-charts@master] stream: mw-page-content-change-enrich
Folks I just did a deployment for mw-content-history-reconcile-enrich to get https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1253633 into prod, but got this unexpected diff:
- image: docker-registry.discovery.wmnet/repos/data-engineering/mediawiki-event-enrichment:v1.43.0 + image: docker-registry.discovery.wmnet/repos/data-engineering/mediawiki-event-enrichment:v1.44.0
I should have followed my instinct but I didn't, and I pushed the change to prod. However, this image upgrade broke production.
I just reverted via 1253650: Revert "stream: mw-content-history-reconcile-enrich" | https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1253650.
I suggest we revert all other production pyflink image changes, and that we test this image change only in -next deployments until we sort the kinks out.
Being bold and reverting changes to mw-page-content-change-enrich to avoid inadvertently repeating T408918#11715866.
1253653: Revert "stream: mw-page-content-change-enrich" | https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1253653
Thanks @xcollazo, I was about to do that deployment today. I'll search logs to understand what happened.
The version 1.44.0 that includes Flink is already deployed on the HTML enrichment pipeline and it works, but we haven't tried with page_content_change yet.
As you suggest, I'll deploy the -next version with the new image and test it, when it works, I'll do the same for the regular deployment.
Change #1254118 had a related patch set uploaded (by JavierMonton; author: JavierMonton):
[operations/deployment-charts@master] stream: mw-content-history-reconcile-enrich-next
Change #1254118 merged by jenkins-bot:
[operations/deployment-charts@master] stream: mw-content-history-reconcile-enrich-next
Change #1254132 had a related patch set uploaded (by JavierMonton; author: JavierMonton):
[operations/deployment-charts@master] stream: mw-content-history-reconcile-enrich
Change #1254135 had a related patch set uploaded (by JavierMonton; author: JavierMonton):
[operations/deployment-charts@master] stream: mw-page-content-change-enrich
Change #1254132 merged by jenkins-bot:
[operations/deployment-charts@master] stream: mw-content-history-reconcile-enrich
Change #1254135 merged by jenkins-bot:
[operations/deployment-charts@master] stream: mw-page-content-change-enrich
The issue was that some time ago, between v1.43 and v1.44 we moved all the code into the python/ folder, so the entry point in Helm needed an update too.
Now all PyFlink applications are on v1.44 and running on prod.