Flink base image should not install into system python environment
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ottomata
	Feb 26 2026, 6:52 PM

Description

Newer versions of python (3.11) & Debian (bookworm) discourage using pip to install packages into system site-packages.

Our WMF built Flink base image installs Flink by pip installing apache-flink into the system site-packages. When it was upgraded to Debian bookworm, this failed, so the --break-system-packages CLI opt was added.

PyFlink application images (like mediawiki-event-enrichment) base on the production Flink image, and use blubber to manage python dependencies and execution. Newer versions of Blubber respect Debian's preference to not install in system site-packages, and create a venv (in /opt/lib/venv). This causes blubber built pyflink apps to break, unless they explicitly set use-system-site-packages.

While this works, the Flink base image should do the right thing and also install into venv (the same one used by blubber built application images).

Done is

Flink base image no longer uses --break-system-packages
PyFlink Application images updated to remove use-system-site-packages from blubber config.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	flink: Install flink in blubber-compatible venv	operations/docker-images/production-images	master	+27 -13

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Update flink base image	repos/data-engineering/mediawiki-event-enrichment!136	atsuko	T418525	main
	python: Allow configuration of the venv path	repos/releng/blubber!171	dduvall	review/use-existing-venv-d45d	main

Customize query in GitLab

Related Objects

Mentioned In: T421216: HTML Enrichment - Tuning & Backfilling configuration
T406872: Fix mediawiki event enrichment to work with newest version of Blubber
Mentioned Here: T406872: Fix mediawiki event enrichment to work with newest version of Blubber

Event Timeline

Ottomata created this task.Feb 26 2026, 6:52 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptFeb 26 2026, 6:52 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Ottomata triaged this task as Low priority.Feb 26 2026, 6:52 PM

Ottomata added a project: Data-Platform-SRE.

Ottomata updated the task description. (Show Details)

dduvall updated https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/171

python: Allow configuration of the venv path

dduvall merged https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/171

python: Allow configuration of the venv path

Nice! https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/171

That will allow us to not have to couple with blubber's default venv path! We'll still use the same one, but if blubber changes it Flink image's will still work!

Ottomata mentioned this in T406872: Fix mediawiki event enrichment to work with newest version of Blubber.Feb 26 2026, 8:04 PM

Ottomata moved this task from Incoming (new tickets) to Backlog on the Data-Engineering board.Mar 4 2026, 5:04 PM

Ottomata moved this task from Backlog to Stream Processing on the Event-Platform board.Mar 16 2026, 2:14 PM

Gehel edited projects, added Data-Platform-SRE (2026-03-27 - 2026-04-17); removed Data-Platform-SRE.Mar 31 2026, 1:20 PM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.

Gehel added a project: good first task.

Thank you for tagging this task with good first task for Wikimedia newcomers!

Newcomers often may not be aware of things that may seem obvious to seasoned contributors, so please take a moment to reflect on how this task might look to somebody who has never contributed to Wikimedia projects.

A good first task is a self-contained, non-controversial task with a clear approach. It should be well-described with pointers to help a completely new contributor, for example it should clearly point to the codebase URL and provide clear steps to help a contributor get set up for success. We've included some guidelines at https://phabricator.wikimedia.org/tag/good_first_task/ !

Thank you for helping us drive new contributions to our projects <3

atsuko claimed this task.Apr 13 2026, 12:41 PM

Looking at docker-pkg, unclear if the Depends: openjdk-17-jre line in images/flink/flink/control is not working as expected, or docker-registry.wikimedia.org/openjdk-17-jre:latest is needed in a config.yaml:base_images.

$ docker-pkg  build images/flink/flink
    FROM {{ "openjdk-17-jre" | image_tag }}
    ^^^^^^^^^^^^^^^^^^^^^^^^^

ValueError: Image docker-registry.wikimedia.org/openjdk-17-jre not found, or it has no tag (known={'docker-registry.wikimedia.org/trixie:latest', 'docker-registry.wikimedia.org/flink:1.20.3-wmf0-20260412', 'docker-registry.wikimedia.org/bookworm:latest', 'docker-registry.wikimedia.org/bullseye:latest'})

atsuko moved this task from Backlog - operations to In Progress on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.Apr 16 2026, 8:51 AM

Change #1272726 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/docker-images/production-images@master] Install flink in blubber-compatible venv

https://gerrit.wikimedia.org/r/1272726

gerritbot added a project: Patch-For-Review.Apr 16 2026, 2:07 PM

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/136

Draft: Update flink base image

Test plan now contains correct invocations for both builds.

@Ottomata shall we move this step to the base package, since the source file is in the base package as well?

copies:
  - from: docker-registry.wikimedia.org/flink:1.20.3-wmf1-20260416
    source: /opt/flink/opt/flink-s3-fs-presto-1.20.3.jar
    destination: /usr/local/lib/python3.11/dist-packages/pyflink/plugins/s3-fs-presto/flink-s3-fs-presto-1.20.3.jar

Change #1272726 merged by Atsuko:

[operations/docker-images/production-images@master] flink: Install flink in blubber-compatible venv

https://gerrit.wikimedia.org/r/1272726

Successfully published image docker-registry.discovery.wmnet/flink:1.20.3-wmf1-20260416

atsuko updated the task description. (Show Details)Apr 17 2026, 11:46 AM

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/136

Update flink base image

Ottomata mentioned this in T421216: HTML Enrichment - Tuning & Backfilling configuration.Apr 17 2026, 2:43 PM

Also it would be good to update search-platform/cirrus-streaming-updater/.pipeline/blubber.yaml, but it needs maven build to make it work. cc @dcausse

Here is proposed diff https://gitlab.wikimedia.org/-/snippets/290

atsuko added a subscriber: dcausse.Apr 17 2026, 4:06 PM

mediawiki-event-enrichment MR is deployed for mw-page-html-content-change-enrich-next (staging) and working just fine.

atsuko closed this task as Resolved.Apr 17 2026, 4:21 PM

atsuko updated the task description. (Show Details)

In T418525#11833996, @atsuko wrote:

Also it would be good to update search-platform/cirrus-streaming-updater/.pipeline/blubber.yaml, but it needs maven build to make it work. cc @dcausse

Here is proposed diff https://gitlab.wikimedia.org/-/snippets/290

Thanks! Uploaded your change and few other things related to the maven build at https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/200

Gehel moved this task from In Progress to Done on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.Fri, Apr 24, 7:47 AM

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.Fri, Apr 24, 7:48 AM

Flink base image should not install into system python environmentClosed, ResolvedPublicActions