Ticket proliferation disambiguration!
- T185233: Modern Event Platform (TEC2) is the overall Modern Event Platform parent ticket.
- T201068: Modern Event Platform: Stream Intake Service is the parent Stream Intake Service task, it describes high level requirements/user stories of this component.
- T201963: RFC: Modern Event Platform: Stream Intake Service is the RFC ticket, it will hopefully be closed once the RFC process finishes.
This ticket will be used to track and task implementation work for the Stream Intake Service.
The RFC in T201963 is still ongoing (waiting for feedback) but as the RFC process takes an undefined amount of time to complete, we should not block on it, and move forward with implementation now.
The Stream Intake Service will be used to intake events over HTTP from both internal and external clients. Those events will be validated and then produced to Kafka. The events API will be compatible (or close to compatible) with the existing eventlogging-service /v1/events API.
- POST of a single event
- POST of an array of events
- Two response modes:
- Fire and forget: HTTP response is given before event is validated and produced to Kafka
- ACKed: HTTP response is given based on event validation status and Kafka produce success
- JSONSchemas of events read from URIs, either local file:/ or remote http:/.
- Schemas for a given URI should be cached
- event schema uris extraced from events, e.g meta.schema_uri field (this should be configurable).
- Destination topics extracted from events
- Configurable topic transformation (e.g. datacenter prefixing)
- Topic schema restriction: only certain schemas should be allowed in certain topics.
- Events that fail for any reason should be produced to Kafka in an error topic with a specific event error schema