As discussed in T228175: Event Platform Client Libraries, we believe we can migrate existent EventLogging extension produced streams to Modern Event Platform components. This will finally allow us to decommission the EventLogging backend pieces:
- varnishkafka-eventlogging
- eventlogging-processor (and eventlog1002)
- meta.wikimedia.org schemas
- refine_eventlogging_analytics
To support existent EventLogging events in eventgate-analytics, we need to do:
- meta.wikimedia.org schemas ported to draft 7 JSONSchema in a git schema repo with common schema included via $ref.
- stream config entry for each (active) EventLogging schema/stream.
- Schema revision extension attributes changed to use the new semver schema version.
- EL client side code adapted to produce full event (with capsule fields) and to POST to eventgate.
- Resolve capsule userAgent type issues (This is a string in JSONSchema, and a struct in Hive)
Ideally, EventLogging will produce the full event including EventCapsule fields to eventgate-analytics-external, the same eventgate instance that new style schemas will use. The same Refine job we use for eventgate analytics events should be able to Refine the old EL style events. Not all fields from capsule will be set (e.g. seqId and recvFrom), but we can work with what we have on the client side. The main issue will be resolving the userAgent type discrepancy, as we will parse the user_agent during refinement.
We'll start by migrating a single high volume EventLogging stream to MEP: VirtualPageView - T238138: Vertical: Virtualpageview datastream on MEP