NOTE: This task captures all of the outputs under the Q1 Product Analytics Infrastructure sub-program of {T225237}.
# Event Platform Client Libraries
Each of our product platforms has an existing library used to detect data in the form of events, and send these events to our [[ https://wikitech.wikimedia.org/wiki/Event_Platform | event platform ]].
- We want to avoid a situation where data from different products cannot be compared due to differences in the way that the data was collected.
- It is impossible to use a single implementation to cover all products.
In order to solve this engineering challenge, we will create a **common specification** for all "event platform clients" to follow. We will then implement the specification, section-by-section, on each of these existing libraries until they are brought into conformance. Once this is done, the specification will serve as a foundation for documentation of the system, and a technical structure on which to build and organize analytics practices across the product organization. In particular, it will provide a guide to re-factoring and upgrading libraries to produce events to [[ https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate | EventGate ]] rather than [[ https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging | EventLogging ]]
#### We are making changes to the client libraries we use to send analytics events.
### SpecificationThe decision to evolve our client libraries arose from a combination of factors.
* {T215435} suggested that these libraries tend to be a source of frustration for product owners, engineers, and analysts.
* {T185233} provided a strong new set of backend capabilities for us to take advantage of, especially {T205319} and {T201063}.
* The results of a [[ https://docs.google.com/presentation/d/178pkMDCckM0RVdVsAO3NFdLqd9H7Uw28b8tcGyk4fkk/edit#slide=id.g55ac32726f_1_93 | stakeholder meeting (slides) ]] in which this project was proposed.
The specification is organized into sections
Evolving our event platform clients (sometimes generally referred to as "EventLogging") in their current state presents a number of challenges. Currently, MediaWiki JavaScript, the Android Wikipedia app, and the iOS Wikipedia app each has some kind of library for sending events to the [[ https://wikitech.wikimedia.org/wiki/Event_Platform | Event Platform ]]. The decision was made to eliminate this heterogeneity as much as possible, by developing a common specification that can be followed by all clients. This would also be beneficial for cross-platform analysis, maintainability, porting, each covering a functional unit of software.testing, These are:and documentation.
- {T230578} - buffering events or HTTP requests### Status
[x] Q1: Use the capabilities of the MEP, along with results from the comprehensive user survey, to define a set of requirements.
- {T230577} - gener[x] Q1: Work backwards from the requirements to a **common specification and formatting of identifier values** that can be followed by all "event platform clients".
- {T230580} - determining whether an instrument is in or out of sample[x] Q1: Implement a prototype of the common specification on each of the three major platforms.
- {T231309} - the stream abstraction and its management[ ] Q2: Work together with analytics engineering to finish developing required MEP components
- {T230579} - tools for developers to inspect behavior during testing
### Implementation[ ] Q2: Work together with product owners and specialists to review, revise, and productionize the prototypes
[ ] Q2: Complete documentation of specification and offer for wide review
Implementation of the specification is#### Platform targeted for the following platforms:ets
- {T228179}
- {T228180}
- {T228181}
#### Code units
**Output controller** - controls when and how events are transmitted over HTTP
**Association controller** - provides, and stores various ID values that associate events at query-time
**Sampling controller** - determines whether a stream is in- or out-sample on the client
**Main** - the public interfaces to add stream configuration and log events
**Debug** - tools for developers to inspect behavior during testing
**Integration** - abstraction layer for external functions, mostly used for testing
More information will be added to this ticket soon as we document the specification.
#### Just, where's the code?
This code is currently undergoing CR, has not yet been moved into Gerrit, and is subject to substantial change.
**Temporary repository:** https://github.com/linehan/wmf-epc
#### Existing client libraries (for reference)
- Browsers:
- Client library core
- https://www.mediawiki.org/wiki/Extension:EventLogging
- https://github.com/wikimedia/mediawiki-extensions-eventlogging
- Instrumentation
- Some called directly using `mw.track("event.<schema>", data)`
- Complex ones found in WikimediaEvents extension:
- https://www.mediawiki.org/wiki/Extension:WikimediaEvents
- https://github.com/wikimedia/mediawiki-extensions-wikimediaevents
- Android:
- Client library core
- https://github.com/wikimedia/apps-android-wikipedia/blob/master/app/src/main/java/org/wikipedia/analytics/EventLoggingService.java
- https://github.com/wikimedia/apps-android-wikipedia/blob/master/app/src/main/java/org/wikipedia/analytics/EventLoggingEvent.java
- https://github.com/wikimedia/apps-android-wikipedia/blob/master/app/src/main/java/org/wikipedia/analytics/Funnel.java
- https://github.com/wikimedia/apps-android-wikipedia/blob/master/app/src/main/java/org/wikipedia/analytics/TimedFunnel.java
- Instrumentation
- Everything else in https://github.com/wikimedia/apps-android-wikipedia/tree/master/app/src/main/java/org/wikipedia/analytics
- iOS:
- Client library core
- https://github.com/wikimedia/wikipedia-ios/blob/develop/Wikipedia/Code/EventLoggingFunnel.m
- https://github.com/wikimedia/wikipedia-ios/blob/develop/Wikipedia/Code/EventLoggingFunnel.h
- Instrumentation
- https://github.com/wikimedia/wikipedia-ios/blob/develop/Wikipedia/Code/EditFunnel.swift and many other such Funnels.