I want to start a discussion about MW Extensions and our intentions with where code should live. Part of this discussion has already been had between me and @Krinkle in comments in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/545400.

In that CR, Timo mentioned concerns about extension proliferation. Product Infrastructure is working on this task to unify and standardize client side libraries within Modern Event Platform. The main one they are working on first is MediaWiki Javascript client side, which is currently implemented by the EventLogging extension. I want to identify how things are now, our ideal (post EventLogging) future, and then we can discuss and make compromises. Ok!

Status Quo

EventLogging

• On wiki JSONSchema namespace and renderer
• Custom JSONSchema validation code
• Client side RL modules for mw.eventLog and mw.track
• Client side debug helpers for event logging

WikimediaEvents

A catch all place for Wikimedia specific instrumentation code. This is a user of EventLogging extension.

Various extension instrumentations

There are various extensions that use the EventLogging extension to log events.

EventBus

Server side MediaWiki specific events. This uses MediaWiki hooks to construct events that are POSTed to an event service (EventGate). Used by e.g. change-prop to re-render pages in RESTBase as well as for MW Job Queue.

Ideal future

Event producer client extension

This would either be a new extension or a replacement for EventLogging. It would have thin MediaWiki Javascript module(s), but contain no server side JSONSchema logic.

(I'll say more about this below.)

This would use the EventStreamConfig extension (below).

WikimediaEvents

This would be used for the same things as it is today.

I'd also consider merging in the code in EventBus to this extension. EventBus and WikimediaEvents seem to be for similiar purposes; a place to emit Wikimedia specific events. This might not be exactly right, as I think @Pchelolo and Marko see EventBus events as a crucial piece of scalable MediaWiki functionality, and want to keep it separate from Wikimedia specific stuff so it is usable by 3rd parties. I'm not sure I agree, but I'd defer to them about this.

This would use the Event producer client extension.

Various extension instrumentations

These would be used for the same things as they are today. These would be users of the Event producer client extension.

EventStreamConfig

This would be a new extension that would contain logic and a MW API endpoint to serve event stream configuration from MW global config. This would be used by

• Event producer client extension to dynamically configure things like sample_rate
• EventGate to enforce that only desired schemas are allowed in certain streams.
• EventBus (or WikimediaEvents?) to figure what event service instance a particular stream should be sent to.
• A potential GUI to help product managers and engineers figure out what streams are currently active and what settings they have.
• Stream Connectors to automate ingestion of schemaed topics into downstream systems. If we can programatically get the schema of any topic, we could use any Kafka Connector with any stream.
• EventStreams (public HTTP streams) could use this to automate stream to topic mappings. A 'stream' is made up of multiple topics (currently one per DC). It would be nice to not have to hardcode this information into EventStreams config.

A stream's config would look something like this:

	[
		'stream' => 'mediawiki.virtual_page_view',
		'schema_title' => 'mediawiki/page/virtual-view',
		'sample_rate' => 0.1,
		'EventServiceName' => 'eventgate-main',
		'topics' => [eqiad.mediawiki.virtual_page_view, codfw.mediawiki.virtual_page_view]
	]

The only new functionality is EventStreamConfig. Everything else 'new' would be intended to replace something that already exists.

There are a few paths to getting to this ideal future. The path of least resistance would be to put everything new into the existent EventLogging extension, including event stream config handling and new client side code APIs.

However, I think Jason and I would prefer to start as clean as possible. That means creating both a new EventStreamConfig extension and a new event producer client extension separate from EventLogging.

We are sensitive to the 'extension proliferation' problem. Even if we have good intentions of replacing EventLogging, we'd likely have to run EventLogging and a new event producer client extension side by side for years as we migrate usages over.

Having 2 separate code paths within the same extension might be confusing to developers and users. EventLogging has a lot of stuff we want to eventually get rid of. It will be easier to do this with a new extension and to migrate usages to it, rather than do so within the same extension.

When we first embarked on an analytics + production event unifcation project in 2015 (the EventBus project), I modified the EventLogging python server side codebase to handle both the new 'eventbus style' events, as well as the original EventLogging capule style events, and to handle their disparate schema repositories. I had intentions of merging these code paths together. This turned out to be a bad idea, and was very confusing. We've since successfully decommissioned the confusing eventlogging-service-eventbus code. We are slowly but surely getting rid of other eventlogging server side functionality. Even if it will be a slow process, I think we can do the same for the EventLogging extension.

@Nuria, I know you are wary of creating a new extension to replace EventLogging. From what I understand, your worry is more about losing the good work that you and Dan and Timo have done in the last year or so to optimize EL and to improve the sampling logic there. I don't think that a new extension would lose this work though; instead the logic you've developed would just be copied over into the new extension.

Ok long discussion over, now to proposals. 2019 Q2's goal is to have working stream config and client side JS code that can use EventGate in MW vagrant, with the intention of deploying to prod in Q3. Let's decide how to proceed!

Event Stream Config extension proposals

• Config-1: New EventStreamConfig extension
• Config-2: Event stream config lives in Event producer client extension (either EventLogging or a new one)

Event producer client extension proposals

• Producer-1: New extension
• Producer-2: EventLogging extension with separate ResourceLoader module, i.e. not ext.eventLogging
• Producer-3: ext.eventLogging module and API (e.g. mw.eventLog, mw.track) augmented to support new usage (POSTing to event gate, no more capsule, users have to construct events, using stream config, etc.)

EventBus extension proposal

• EventBus-1: Merge EventBus into WikimediaEvents.
• EventBus-2: Leave EventBus as is.

We don't have to discuss this proposal now, as we wouldn't do this anytime soon, and it doesn't block the goals for this or next quarter.

@Nuria, @Milimetric, @jlinehan, @Krinkle, @Pchelolo, @Neil_P._Quinn_WMF please opine! (I'll give my preferences in next comment.)

My preferences:

• Strong preference for Config-1.
• In favor of Producer-1. Producer-2 is ok too. Don't want Producer-3.
• Prefer EventBus-1, but I expect Petr to convince me otherwise :)

Also @mpopov

@Nuria, I know you are wary of creating a new extension to replace EventLogging. From what I understand, your worry is more about losing the good work that you and Dan and Timo have done in the last year or so to optimize EL and to improve the sampling >logic there. I don't think that a new extension would lose this work though; instead the logic you've developed would just be copied over into the new extension.

Given that we would be having two systems for quite some time I do not think is wise to have this code duplicated, for logistics and pef reasons. It is very likely we still need to stop supporting EL in a year. So you are looking at having critical code duplicated for a long time. Also I do not agree with the argument that "there is a lot we want to get rid of in the EL extension", the transport, enqueing and sampling remains so the majority of the code will actually remain. The capsule and couple other small bits will be removed but this is actually not such a large change. Functionality around posts will need to be added. Overall, changes are not that dramatic.

The code we use to log events should be light enough that can be included on every page, this is an argument against duplicating as well, we want to maintain the ability to post to both systems in ALL pages (including first pageload) for the time we need to migrate from one system to the other one.

In a more phylosophical note I think rewrites that start afresh "generate the illusion of progress" while not providing immediate value, we do not need to start fresh to evolve the system we have if it is designed to be maintainable. In this case I do see any barriers towards evolving the EL extension.

the transport, enqueing and sampling remains so the majority of the code

https://github.com/wikimedia/mediawiki-extensions-EventLogging/tree/master/includes is a lot of code.
The transport is different, POST vs html encoded GET query params, no URL length limit.

In a more phylosophical note I think rewrites that start afresh "generate the illusion of progress"

Perhaps, but it also allows for a clearer software lifecycle and separations of concerns.

The code we use to log events should be light enough that can be included on every page, this is an argument against duplicating as well, we want to maintain the ability to post to both systems in ALL pages

The only code in https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/modules/ext.eventLogging/core.js that will be possibly reusable are the sampling functions near the bottom. Perhaps we can factor these functions out of ext.eventLogging core.js into a separate RL module usable by a ext.eventLogging and either a new extension or just a new RL module in EventLogging for event platform stuff.

@Nuria, could you indicate your preferences for each of the proposals, and how strongly you feel about each one?

The only code in https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/modules/ext.eventLogging/core.js that will be possibly reusable are the sampling functions near the bottom.

I was thinking of sendBeacon code when I mentioned transport , as, as you noted, ability to do POSTs needs to be added.
So this code does not change much https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/modules/ext.eventLogging/core.js#L121

Incoming code that has to do with queueing also remains: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventLogging/+/524575/
As well as bugfixes like: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EventLogging/+/14b61af18e1655d0322fd15655300b4dbc0339b5/modules/ext.eventLogging/subscriber.js#57

and also ability to debug events client side, which might change slightly : https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/modules/ext.eventLogging.debug/debug.js

And really, EL doesn't have much more than that. It is pretty small. I look at the event-gate client as adding ability to do posts and source config.

I think my thoughts align with what you describe here: Config-1, Producer-3 but Ia m not sure I understand what you are proposing on Producer-2.

I think my thoughts align with what you describe here: Config-1, Producer-3 but Ia m not sure I understand what you are proposing on Producer-2.

Producer-2 proposal is to put the new code we need in a separate ResourceLoader Javascript module in EventLogging e.g 'ext.eventLogging.producer' or whatever. This would at least let us create a new API interface other than mw.eventLog. Perhaps mw.eventLog (in ext.eventLogging) could be made to proxy to the new code paths in the right cases.

In T228175#5606900, @Ottomata wrote:

My preferences:

• Strong preference for Config-1.
• In favor of Producer-1. Producer-2 is ok too. Don't want Producer-3.
• Prefer EventBus-1, but I expect Petr to convince me otherwise :)

My preferences are the same.

But I'll talk some more about Producer-* since it mostly concerns code that I've been writing with @mpopov. Reading the discussion, I think I can clarify some things before adding my own opinions about the options.

The new client code has already been written (no wheels were re-invented or harmed)

This was part of our Q1 goals. There are equivalent implementations for Java and Swift.

I admit the tickets are organized strangely and didn't get updated frequently or at all, so this might not have been clear. (We were using the tickets as simple Kanban cards on the Better Use of Data board).

In T228175#5607056, @Nuria wrote:

In a more phylosophical note I think rewrites that start afresh "generate the illusion of progress" while not providing immediate value, we do not need to start fresh to evolve the system we have if it is designed to be maintainable.

I agree and there is no shortcut to code that has been battle-tested in production. That's why, if it was a capability that EventLogging currently does, we incorporated that behavior wholesale.

What is the same:

1. Random identifiers are generated using mw.user.generateRandomSessionId(), the exact same as in EventLogging
2. The sampling algorithm in the prototype is identical to randomTokenMatch from EventLogging
3. An equivalent algorithm and data structures to the one that @Milimetric has developed for https://phabricator.wikimedia.org/T225578 is used for output buffering.
4. Identical use of mw.track and mw.trackQueue for input buffering.
5. mw.track('event.<whatever>', data) is still the general call format.

Notice that, as @Ottomata pointed out earlier, the only things here that are actually part of the current extension (as opposed to behavior or integration) is the sampling function.

What is new:

• Requests are using HTTP POST
• True session identifiers (persisted across pageviews)
• Activity identifiers (tag all events comprising an individual run of a funnel or feature)
• Stream configuration loading
• Stream configuration awareness, including:
  • Whether the stream is active (can receive events)
  • Whether the stream can receive trackable info or not
  • Sampling rate
  • Intake URL (e.g. which EventGate instance)
• Multiple dispatch of events: event for stream x will be copied to streams matching x.*
• Application can enable/disable transmission of output buffer, so connection loss, sleep modes, etc. can happen without losing events.

Putting some numbers on the size discussion

In T228175#5607056, @Nuria wrote:

The code we use to log events should be light enough that can be included on every page, this is an argument against duplicating as well, we want to maintain the ability to post to both systems in ALL pages (including first pageload) for the time we need to migrate from one system to the other one.

I agree again. But here, the code that would overlap between the two could actually be very small. So, I think that the difference in size between a single extension with two code paths, and two standalone extensions, could be small. If it is a concern, we could measure it.

I'll share some numbers on the size of the prototype in relation to the EventLogging extension:

Note that the prototype client is smaller than what we have in production right now, despite its additional features. So it's not very onerous.

Why I prefer Producer-1

The reason that Producer-1 appeals to me has not changed since we first discussed it at the end of the last FY. It's because migration seems easier. We could go extension-by-extension switching from EventLogging to the new extension, until none of them are requiring EventLogging. It would be simple to divide into work units, and it would be easier to sunset EventLogging as a whole extension that way, because we know where it isn't being called.

If the concern is about the size in production, my feeling is that no matter which option we choose, there is going to be more code getting pushed down during the migration period, and that the amount of surplus will not change significantly between the three approaches.

But still, let's assume that the new extension in Producer-1 is like the current prototype, i.e., that it is smaller than the current EventLogging extension. Then, the average payload size over the course of the migration effort during Producer-1 is likely to be smaller than Producer-3, since more and more pages would be loading with only one of the extensions, and it would have a smaller size.

This could also perhaps be said of Producer-2, but why wouldn't we rather have two small, focused, well-documented extensions than one that has two answers to every question?

My intuition is that it will be tricky trying to mix existing code with new code in a way that preserves the current behavior alongside the new behavior. In the absence of some kind of benefit, I don't really see why we would risk it? This has already happened in @Ottomata 's adventures with EventBus and EventLogging, and I've experienced it myself on other projects.

Maybe there are advantages of Producer-2 or Producer-3 that I am not thinking about?

Several things that come to mind:

I think the list of features that would be included in a 1st implementation is a bit ambitious. This one for example:

• Activity identifiers (tag all events comprising an individual run of a funnel or feature)

requires significant changes on mediawiki client side (robust per device storage caching and eviction in a number of browsers) that to be honest I find hard to believe will get done and throughly tested to be deployed to prod in the medium term.

Ditto for this one which is a lot more complicated to get right:

• Application can enable/disable transmission of output buffer, so connection loss, sleep modes, etc. can happen without losing events.

let's assume that the new extension in Producer-1 is like the current prototype, i.e., that it is smaller than the current EventLogging extension

The differences in this code on a prototype and something that gets deployed to the majority of the user base are going to be very significant, I could not see a link to prototype in this ticket but if you add that one we can get into details.

We could go extension-by-extension switching from EventLogging to the new extension, until none of them are requiring EventLogging.

I think this is where we disagree, the EventLogging extensions is just a facade. The core functionality is being extended but the basic functionality has not changed, as such we would expect a level of backwards compatibility and in large enough systems is always a safer strategy doing migrations that are (specially initially) completely transparent to the clients. I find a very likely scenario in the WMF that we deploy two extensions with vastly different apis and in the middle of the migration happening resources get shifted around and then we have duplicated code that nobody owns in prod, two ways to send analytics events in our codebase and a half-done non-backwards compatible migration.

Ah, sorry I see link to prototype code here: https://github.com/linehan/wmf-epc/blob/jason/src/js/epc.js

In T228175#5624465, @Nuria wrote:

• Activity identifiers (tag all events comprising an individual run of a funnel or feature)

requires significant changes on mediawiki client side (robust per device storage caching and eviction in a number of browsers) that to be honest I find hard to believe will get done and throughly tested to be deployed to prod in the medium term.

Yes, same as with session identifiers, the same persistence mechanism is used (whatever it might be). As we've spoken about this before, there are only really two approaches to this persistence in the browser (localStorage or cookies), and it's a determination to make with the performance team, etc. I'm just noting that these are things which the current code, as written, does, and that a production version would ultimately aim to do (after all the decisions about where the persistence happens are made).

Ditto for this one which is a lot more complicated to get right:

• Application can enable/disable transmission of output buffer, so connection loss, sleep modes, etc. can happen without losing events.

The library itself is not determining when to enable or disable sending. It provides a way for the application to signal that, that's all. Whether people would want to take advantage of it is up to the application maintainer, and getting it right is also a problem for them to solve. The library will just do what they tell it to do.

I could not see a link to prototype in this ticket but if you add that one we can get into details.

It's hiding up there (maybe blue-on-blue is hard to see). Here's the link: https://github.com/linehan/wmf-epc/blob/jason/src/js/epc.js

the EventLogging extensions is just a facade. The core functionality is being extended but the basic functionality has not changed

Aah, I think I understand now! So whenever I talk about a "migration," you are saying, "what migration?" I see, I see. Well, I don't know. I think that it wouldn't be so cleanly backwards-compatible, but maybe I am wrong? Maybe it's something we should actually roll up our sleeves and try out.

In terms of resources, I know that at least myself and @mpopov are resourced through the end of the FY to work exclusively on this project, so I'm confident we'll be able to see it through until then at least. Although, I can understand the concern about ending up stranded in the intermediate state. That goes for the other proposals above too, where we are going to potentially have 5 extensions before we have 3.

The core functionality is being extended but the basic functionality has not changed, as such we would expect a level of backwards compatibility and in large enough systems is always a safer strategy doing migrations that are (specially initially) completely transparent to the clients.

We don't have a plan for how we would accomplish this. IIRC, Nuria had an idea around deploying another EventGate instance that could somehow replace varnishkafka-eventlogging + eventlogging-processor. Then we'd somehow modify mw.eventLog in some backwards compatible way that would POST the existing events as they are to this new EventGate instance. That would certainly let us decommission some old backend components, but I'm not sure it addresses some of the bigger goals of MEP, e.g. getting rid of the centralized on wiki schema repository. Perhaps there is a way to do this all, but we haven't thought of one yet.

Should we spend some time brainstorming ways we could be backwards compatible with existent EventLogging schemas? I had planned not to be on purpose, but maybe there are ways.

This could also perhaps be said of Producer-2, but why wouldn't we rather have two small, focused, well-documented extensions than one that has two answers to every question?

I think an argument against Producer-1 is 'extension proliferation'. There's just more to maintain and deploy and get working in MW vagrant and other dev environments if there are more extension dependencies.

Ideally I'd prefer Producer-1, but perhaps Producer-2 is acceptable. From ResourceLoader's perspective, Producer-1 and Producer-2 are identical; user extensions of the event producer specify dependencies at the ResourceLoader module level. Producer-1 would have ext.eventProducer (or whatever) and Producer-2 would have ext.eventLogging.NewClientBlabla (instead of ext.eventLogging).

In T228175#5624514, @Ottomata wrote:

From ResourceLoader's perspective, Producer-1 and Producer-2 are identical

Yeah, I agree it's the same, which is why I'm not sure I understand the argument about proliferation (in the case of Producer-*) except in the case where there is abandonment of this project in the intermediate state. My version of @Nuria's concern about intermediate states is that I worry about an intermediate state where we have a bunch of dead (or barely-used) modules in the extension, or people still think EventLogging does X when it now does Y, etc.

If the overhead of a new extension is going to outweigh the overhead of mingling old and new code, then I can sort of understand. However we get there, it seems like we need to spend some time formulating exactly what the end state is going to be -- like @Ottomata mentioned, getting rid of the on-wiki schema repo stuff, deprecating GET support, etc.

Talking some more with @Ottomata on IRC, we agreed we ought to brainstorm ways we can do this like a facade. We have a couple ideas already, maybe it's possible. @Ottomata is off tomorrow and I'm at the Product Analytics offsite next week, but I'll set a meeting for after that. Keep leaving thoughts here, no point in waiting a week if you have a good idea.

The library itself is not determining when to enable or disable sending. It provides a way for the application to signal that, that's all.

Let's talk more about this, in this case (offline/online) the eventlogging client IS the application that handles responses to a disconnect. Just making a note of this but discussion on this regard can happen later, in fact, i do not think a first instance should tackle this problem.

In T228175#5624674, @Nuria wrote:

Let's talk more about this, in this case (offline/online) the eventlogging client IS the application that handles responses to a disconnect. Just making a note of this but discussion on this regard can happen later, in fact, i do not think a first instance should tackle thsi problem.

Sounds good. Here's a simple example of how you might use it (imagine that EventLogging is the name of the library module):

/* Let's all briefly imagine a world where 'online' and 'offline' work properly in all browsers */

window.addEventListener('offline', function() { 
        EventLogging.disable_sending();
});

window.addEventListener('online', function() { 
        EventLogging.enable_sending();
});

I'm thinking of things like the proposed SessionLength event, where you'd have a gap in the ping counter due to a loss of network connectivity, stuff like that. Since we have the buffer there anyway. Well, getting rid of it would make the code even simpler, so I'm not against it. Let's talk more!

I updated the description and moved the many tiny confusing subtasks to T237106: EPC Kanban Cards.

Thanks Jason!

In prep for our meeting tomorrow to see if we can be backwards compatible with EventLogging events, I started brainstorming on our notes etherpad:
https://etherpad.wikimedia.org/p/event-platform

I think we can do this, but it isn't easy. I'd like to be able to do this without deploying another eventgate instance just for backwards compatibility, and I would like to eventually get rid of: schemas on meta.wikimedia.org, varnishkafka-eventlogging, and eventlogging-processor. To do this, we need:

• meta.wikimedia.org schemas ported to draft 7 JSONSchema in a git schema repo with common schema included via $ref.
• stream config entry for each (active) EventLogging schema/stream.
• Schema revision extension attributes changed to use the new semver schema version.
• EL client side code adapted to produce full event (with capsule fields) and to POST to eventgate. Hopefully we can keep the client side code differences between old EL events and new MEP ones minimal.

I'm not sure how to solve the apps problem though. They make GET requests to produce EventLogging events, and we can't force installed apps to upgrade.

Let's discuss more tomorrow :)

Nice, this is a good start.

I'm not sure how to solve the apps problem though. They make GET requests to produce EventLogging events, and we can't force installed apps to upgrade.

We can deal with this later, the solution here really depends from the volume of valid events comings from those apps and its criticality, There are a few strategies. For now let's focus on a clean start for apps (via new client) and a backwards compatible js event.

Just saw this ticket on passing, maybe we want to rework the timeline/text of ticket ? (as deliverables were set for last year 2019/2020)

In T228175#6363685, @Nuria wrote:

Just saw this ticket on passing, maybe we want to rework the timeline/text of ticket ? (as deliverables were set for last year 2019/2020)

Agree, will be updating soon.