Page MenuHomePhabricator

Support topics without a schema in Flink Catalog
Open, Needs TriagePublic

Description

In the current implementation of the catalog, an event stream and its associated schema must exist for a table to be created. However, there might be some instances where you would want to write to a topic that doesn't have a schema, or even borrow a schema from a different event stream.

There should be some options to allow this, prefixed with unsafe so people know what they're getting into.

'unsafe.no-validation'='true',
'unsafe.schema-from'='test.event.schema'

Having no validation would probably also mean disabling the event normalization completely. There might also have to be some guardrails in place so that these options can't be used on event streams found in the event stream config.

Event Timeline

What is the case where someone would need to be able to write without a schema? I think, if there is no JSONSchema, then they'd have to use the default Flink Kafka connector CREATE TABLE stuff and specify the schema themselves. This is all possible now using the defaultt in Kafka connector.

But, I def see a use case where someone might want to produce with a JSONSchema to an arbitrary Kafka topic, mostly for debugging and/or backfilling. An option to allow this would be good. Maybe:

'event-stream-name.check.enabled' = 'false',
`event-schema-uri' = '/test/event/1.1.0'

?