Page MenuHomePhabricator

Declare webrequest as an Event Platform stream
Open, Needs TriagePublic


webrequest is not technically an 'Event Platform' stream. Making it one would allow us to use tooling we are developing as part of T308356: [Shared Event Platform] Ability to use Event Platform streams in Flink without boilerplate to consume webrequest from Kafka using Flink, or any other event platform tooling. This would be nice for {T310997}.

To make webrequest supported by Event Platform we need the following:

  • An event schema declared that matches the webrequest fields.
  • The following fields added to webrequest's varnishkafka output format:
  • $schema
  • (this can be just set to 'webrequest')
  • A 'webrequest' stream declared in event stream config, with its composite topics explicitly declared

Once these are done, we should be able to treat webrequest like any other event stream. (We could also use event stream discovery in gobblin for webrequest ingestion, if we wanted to!)