Eventutilities Flink: port SerDe tests from SUP
Open, HighPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	pfischer
	Sep 15 2025, 2:53 PM

Description

This is a follow-up of T372912

hit by a nasty problem where after upgrading to flink-1.20 we forgot to update eventutilities to 1.4, this caused the RowSerializer to be used in the flink state and the job to fail upgrading. Reason is that the createSerializer function has changed its signature and running flink 1.20 without the new version of EventRowTypeInfo in 1.4 flink called the parent class RowTypeInfo that returns a RowSerializer. This is particularly fragile and we should probably stop extending RowTypeInfo.

As discussed by the SIG on 24.09.25, we want to prevent this by porting the backwards compatibility tests for snapshots (and their serialisers) from SUP to eventutilities. As an additional layer of safety, SUP (and other consumers of eventutilities) should enforce compatibility at dependency resolution level, for example, by using enforcer, see T405591.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	flink: test schema evolution	wikimedia-event-utilities	master	+44 -4
	flink: Add snapshot tests	wikimedia-event-utilities	master	+218 -0

Customize query in gerrit

Related Objects

Mentioned In: T405591: SUP: Enforce coherent flink version in transitive dependencies
T372912: Migrate image recommendation to use page_weighted_tags_changed stream
Mentioned Here: T405591: SUP: Enforce coherent flink version in transitive dependencies

Event Timeline

pfischer created this task.Sep 15 2025, 2:53 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 15 2025, 2:53 PM

pfischer mentioned this in T372912: Migrate image recommendation to use page_weighted_tags_changed stream.Sep 15 2025, 2:57 PM

Gehel triaged this task as High priority.Sep 15 2025, 3:13 PM

Gehel edited projects, added Discovery-Search (2025.09.05 - 2025.09.26); removed Discovery-Search.

Gehel added a project: Essential-Work.Sep 15 2025, 3:24 PM

Gehel added a project: Data-Engineering.Sep 15 2025, 3:28 PM

Ottomata subscribed.Sep 16 2025, 1:57 PM

Ottomata added a project: Event-Platform.Sep 17 2025, 6:46 PM

Ottomata moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.

Ottomata added a project: Data-Engineering-Radar.

@Ottomata, we were hit by a breaking change in Flink's API when upgrading to 1.20. In order to make our code more resilient, @dcausse suggested we should stop relying on RowTypeInfo for (de-)serialisation. Since most of the related code is part of eventutilities, I would appreciate your feedback regarding that approach.