Page MenuHomePhabricator

Control instruments with Stream Configuration
Open, Stalled, MediumPublic

Description

This task tracks work for a demonstration of the new capabilities for creating datasets that the client libraries enable. The demo will cover combinations of basic operations using examples drawn from past experience, to illustrate how it can save time. Operations covered would be

  1. Placing instrumentation in code
  2. Defining a stream configuration
  3. Deploying a stream configuration
  4. At least the following stream configuration operations
    1. Multiple streams consuming the same instrument
    2. One stream consuming multiple instruments
    3. Streams requesting metadata and having it automatically included in the event
    4. Streams specifying targeting using value-based sampling

Specifics to be added in near future.

Event Timeline

jlinehan triaged this task as Medium priority.Feb 8 2021, 7:28 PM
jlinehan moved this task from Inbox to Doing on the Better Use Of Data board.

Change 663211 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[mediawiki/extensions/EventLogging@master] [WIP] Streams subscribe to one or more instruments

https://gerrit.wikimedia.org/r/663211

Change 663616 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventLogging@master] MEP: Add stream config support for filtering by value

https://gerrit.wikimedia.org/r/663616

Change 661975 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventLogging@master] MEP: Demo support for adding login state and wiki db name via config

https://gerrit.wikimedia.org/r/661975

I would love to use the mechanism described in 4.C:

Streams requesting metadata and having it automatically included in the event

For T210106, I would add the ability to transparently fill a user_edit_count_bucket field. But maybe it's too early, the deployed code can only write to a test object?

Also, the existing schemas have a variety of property paths into which this data should land, for example TemplateWizard wants the data in event.performer.user_edit_count_bucket, but QuickSurveysResponses wants it in event.editCountBucket. Will the final design include this kind of flexibility?

kzimmerman moved this task from Doing to To Do on the Better Use Of Data board.

Change 663616 abandoned by Mholloway:
[mediawiki/extensions/EventLogging@master] MEP: Add stream config support for filtering by value

Reason:
There was a big structural change in core.js, and also the syntax for this is now completely different. I'll start a new patch.

https://gerrit.wikimedia.org/r/663616

Change 661975 abandoned by Mholloway:
[mediawiki/extensions/EventLogging@master] MEP: Demo support for adding login state and wiki db name via config

Reason:
Will start a new patch for this rather than rebasing.

https://gerrit.wikimedia.org/r/661975

Change 673345 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventLogging@master] [WIP] Metrics Platform: Filter events by value

https://gerrit.wikimedia.org/r/673345

Change 674620 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventLogging@master] [WIP] Metrics platform: Add context values requested in stream config

https://gerrit.wikimedia.org/r/674620

Change 698018 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] Allow curating stream data sets based on event data values

https://gerrit.wikimedia.org/r/698018

Change 673345 abandoned by Mholloway:

[mediawiki/extensions/EventLogging@master] Metrics Platform: Filter events by value

Reason:

See I2afae4799cb0fdcf26bdf75f4c3621c80a95c069

https://gerrit.wikimedia.org/r/673345

Change 674620 abandoned by Mholloway:

[mediawiki/extensions/EventLogging@master] Metrics Platform: Add context values requested in stream config

Reason:

See I8ab62c8ac63bda9e91c8dacba18fd805de655c77

https://gerrit.wikimedia.org/r/674620

Change 704630 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] [Java] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/704630

Change 698241 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] [JS] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/698241

Change 705435 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] [PHP] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/705435

Change 709820 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] [WIP] [Java] Support curating stream data sets based on event properties

https://gerrit.wikimedia.org/r/709820

Mholloway renamed this task from [Client libraries] Demonstrate new capabilities to Control instruments with Stream Configuration.Aug 11 2021, 1:43 PM

Change 713927 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] [WIP] [Swift] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/713927

Change 716057 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] [VERY WIP] [Swift] Support curating stream data sets based on event properties

https://gerrit.wikimedia.org/r/716057

Change 719378 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[mediawiki/libs/metrics-platform@master] [PHP] Support curating stream data sets based on event properties

https://gerrit.wikimedia.org/r/719378

Change 704630 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [Java] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/704630

Change 709820 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [Java] Support curating stream data sets based on event properties

https://gerrit.wikimedia.org/r/709820

Change 713927 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [Swift] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/713927

Change 716057 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [Swift] Support curating stream data sets based on event properties

https://gerrit.wikimedia.org/r/716057

Change 698241 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [JS] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/698241

Change 698018 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [JS] Support curating stream data sets based on event properties

https://gerrit.wikimedia.org/r/698018

Change 705435 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [PHP] Support adding context values based on stream config

https://gerrit.wikimedia.org/r/705435

Change 719378 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [PHP] Support curating stream data sets based on event properties

https://gerrit.wikimedia.org/r/719378

DAbad changed the task status from Open to Stalled.Wed, Oct 6, 2:10 PM
DAbad moved this task from Work in Progress to Blocked on the Metrics-Platform board.
DAbad added a subscriber: DAbad.

Moving to blocked until we are ready to deploy for end-to-end testing