Page MenuHomePhabricator

Add CI step to event schema repositories to test to fail if a schema is deleted
Open, Needs TriagePublic

Description

In https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/13, a schema was deleted before its stream config was removed. This caused canary events and Hive ingestion jobs to fail, as they tried to look up the schema for the stream.

Our documentation states

A schema should never be deleted, but all of the stream related code and configuration can be removed at anytime to stop producing an event stream

We should add a test to jsonschema-tools, or just to gitlab CI, that fails if schemas are deleted.

This might be difficult to do in jsonschema-tools, as it would need to inspect git history.