Page MenuHomePhabricator

EventLogging MEP Upgrade Phase 1
Closed, ResolvedPublic

Description

The new event platform client library features will be folded into the existing EventLogging extension in phases. This is Phase 1.
For starter code, see: https://github.com/linehan/el-epc-patch-phases.

Suggest changes to the API and requirements and update this description accordingly.

Phase 1
Suggested API

  • produce( string streamName, object eventData )
  • configure( object streamConfig )

Requirements

  • Client can be configured with stream configuration objects
  • Client sends events via HTTP POST
  • Callers do not specify meta.dt, meta.id, or meta.stream
  • meta.dt must be set at the time produce() is first called (i.e., client-side)
  • If produce() is called with data containing meta.dt, that value cannot be overwritten
  • An empty eventData object can be passed to produce()
  • After initial loading, stream configuration cannot be updated during runtime
  • The provided stream configuration is not adjusted by the client during loading (e.g. defaults)
  • produce() does not modify the values in eventData

Open questions

  • Does MediaWiki needs a reliable way to generate ISO 8601 timestamps for meta.dt?
  • How do instruments provide the $schema field?
  • Do we allow stream configuration to specify a per-stream destination URL?
  • Do we support multiple calls to configure() during initial loading?
    • If so, how do we resolve collisions?

Event Timeline

jlinehan created this task.Feb 6 2020, 6:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 6 2020, 6:59 PM
mforns added a comment.Feb 7 2020, 4:43 PM

Thanks a lot @jlinehan for refactoring the older task and putting together this one.

jlinehan updated the task description. (Show Details)Feb 7 2020, 6:07 PM
fdans triaged this task as High priority.Feb 10 2020, 5:24 PM
fdans added a project: Analytics-Kanban.
fdans moved this task from Incoming to Event Platform on the Analytics board.

Change 573677 had a related patch set uploaded (by Mforns; owner: Mforns):
[mediawiki/extensions/EventLogging@master] [WIP] Add ext.eventLogging.client module, phase 1

https://gerrit.wikimedia.org/r/573677

Ottomata added a comment.EditedFeb 21 2020, 4:42 PM

OK @jlinehan @mforns, it's bikeshed time.

I think mw.eventClient is not a great name. Client of ...event?

I suggested to Marcel that we use mw.eventProducer (and 'ext.eventLogging.producer' RL module), but perhaps that name doesn't fit well if we consider the code from the other phases in specified in T238544: EventLogging MEP Upgrade.

Marcel just suggested 'eventStreams'? mw.eventStreams.configure()? Not bad.

Also, am looking at EL RL package modules more.

We originally wanted a new RL package module to avoid activating this new stuff on every page load, but we'll likely enable use this in WikimediaEvents anyway which IIUC is basically every page load already.

Suggestions:

Put this new code (i.e. configure() and produce() functions) into ext.eventLogging in maybe 'client.js', or 'producer.js' or 'eventstreams.js' (dependent on name bikeshed above).

Then, we attach the object or the functions directly to mw. Ideas:

  • Attach object to mw.eventLog.client: mw.eventLog.client.produce()
  • Stick the object on mw directly: mw.eventProducer.produce() or maybe mw.eventStreams.produce()
  • Attach only the functions to mw.eventLog: mw.eventLog.produce(...)

Thoughts?

First: name of file & class & singleton

I think mw.eventStreams is cool!

Second: RL package module organization

Will this new client ever produce events that do not go to the EventLogging pipeline?
If yes, then I think we should create another extension, like: ext.eventStreams (or whatever name we choose).
If not, and you think the name EventLogging will prevail forever, then we can keep it inside ext.eventLogging.

From the 3 options of mw. Ideas, I like the second one the most:

Stick the object on mw directly: mw.eventProducer.produce() or maybe mw.eventStreams.produce()

If not, and you think the name EventLogging will prevail forever

EventLogging will prevail forever, as the name of this extension. It will come to mean JUST the name of this extension though, there will be no more 'EventLogging pipeline'.

Change 574077 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Bump eventgate dependency to 4cd2eaa to pick up parsing for text/plain content type

https://gerrit.wikimedia.org/r/574077

Change 574077 merged by Ottomata:
[eventgate-wikimedia@master] Bump eventgate dependency to 4cd2eaa to pick up parsing for text/plain content type

https://gerrit.wikimedia.org/r/574077

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Mar 2 2020, 3:52 PM

Change 580466 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[integration/config@master] EventLogging now depends on EventStreamConfig

https://gerrit.wikimedia.org/r/580466

Change 580466 merged by jenkins-bot:
[integration/config@master] EventLogging now depends on EventStreamConfig

https://gerrit.wikimedia.org/r/580466

Change 581040 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] Add EventStreamConfig to gated extensions(!) as EventLogging depends on it

https://gerrit.wikimedia.org/r/581040

Change 581040 merged by jenkins-bot:
[integration/config@master] Add EventStreamConfig to gated extensions(!) as EventLogging depends on it

https://gerrit.wikimedia.org/r/581040

Mentioned in SAL (#wikimedia-releng) [2020-03-18T18:41:34Z] <James_F> Zuul: Add EventStreamConfig to gated extensions T244521

Change 573677 merged by Ottomata:
[mediawiki/extensions/EventLogging@master] Add eventStream (phase 1) to ext.eventLogging

https://gerrit.wikimedia.org/r/573677

mforns moved this task from In Progress to Done on the Analytics-Kanban board.Mar 31 2020, 2:57 PM
Nuria closed this task as Resolved.Apr 27 2020, 8:29 PM