Flink Tables should have a default ROWTIME column.
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	gmodena
	Nov 30 2022, 7:56 PM

Description

A number of SQL functions (e.g. window aggregation) assume that the table has a timestamp column tagged as ROWTIME. AFAIK it’s something that can be specified either in the DDL or watermaking strategy.

Summary of a discussion I had with @Ottomata. This information could be provided:

By the Catalog.
In the defaults in EventTableDescriptorBuilder in setupKafka

We have some defaults in 2. for setting the watermar which probably should use the same column for ROWTIME. Default being the kafka timestamp

Event Timeline

gmodena created this task.Nov 30 2022, 7:56 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptNov 30 2022, 7:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Dec 1 2022, 2:12 PM

gmodena claimed this task.Dec 5 2022, 9:32 AM

gmodena moved this task from Backlog to To be Estimated/To be discussed on the Event-Platform board.

JArguello-WMF edited projects, added Event-Platform (Sprint 05); removed Event-Platform.Dec 5 2022, 2:08 PM

gmodena moved this task from Next Up to In Progress on the Event-Platform (Sprint 05) board.Dec 5 2022, 2:53 PM

According to the doc we can declare a rowtime field either via DDL or when converting DataStream -> Table.

Eventutilities already has the capability to add a watermark column when building a TableDescriptor. The column carries "event time" Flink semantics (rowtime) as described in https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/concepts/time_attributes/.

For this reason I went with option 1 and propose a change to the Catalog's getTable() method to add watermark metadata at run time.
MR at https://gitlab.wikimedia.org/tchin/flink-wmf-event-catalog/-/merge_requests/1/edit.

gmodena moved this task from In Progress to In Review on the Event-Platform (Sprint 05) board.Dec 5 2022, 7:31 PM

For this reason I went with option 1 and propose a change to the Catalog's getTable() method to add watermark metadata at run time.

Makes sense!

JArguello-WMF moved this task from In Review to Done on the Event-Platform (Sprint 05) board.Dec 7 2022, 2:09 PM

JArguello-WMF closed this task as Resolved.Jan 9 2023, 2:20 PM

JArguello-WMF moved this task from Sprint 05 to Sprint 07 on the Event-Platform board.

JArguello-WMF edited projects, added Event-Platform (Sprint 07); removed Event-Platform (Sprint 05).

Flink Tables should have a default ROWTIME column.Closed, ResolvedPublicActions

Description

Event Timeline

Flink Tables should have a default ROWTIME column.
Closed, ResolvedPublic
Actions