Page MenuHomePhabricator

Decommission EventLogging backend components by migrating to MEP
Open, MediumPublic

Description

As discussed in T228175: Event Platform Client Libraries, we believe we can migrate existent EventLogging extension produced streams to Modern Event Platform components. This will finally allow us to decommission the EventLogging backend pieces:

To support existent EventLogging events in eventgate-analytics, we need to do:

  • meta.wikimedia.org schemas ported to draft 7 JSONSchema in a git schema repo with common schema included via $ref.
  • stream config entry for each (active) EventLogging schema/stream.
  • Schema revision extension attributes changed to use the new semver schema version.
  • EL client side code adapted to produce full event (with capsule fields) and to POST to eventgate.
  • Resolve capsule userAgent type issues (This is a string in JSONSchema, and a struct in Hive)

Ideally, EventLogging will produce the full event including EventCapsule fields to eventgate-analytics-external, the same eventgate instance that new style schemas will use. The same Refine job we use for eventgate analytics events should be able to Refine the old EL style events. Not all fields from capsule will be set (e.g. seqId and recvFrom), but we can work with what we have on the client side. The main issue will be resolving the userAgent type discrepancy, as we will parse the user_agent during refinement.

We'll start by migrating a single high volume EventLogging stream to MEP: VirtualPageView - T238138: Vertical: Virtualpageview datastream on MEP

Event Timeline

Ottomata created this task.Wed, Nov 13, 4:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Nov 13, 4:37 PM
Ottomata triaged this task as High priority.Thu, Nov 14, 6:26 PM
Ottomata lowered the priority of this task from High to Medium.
Ottomata moved this task from Incoming to Modern Event Platform on the Analytics board.