Page MenuHomePhabricator

eventgate-wikimedia should expose runtime stream configuration
Closed, ResolvedPublic

Description

Some eventgate instances use static stream config, others use remote dynamic stream config (via the MW EventStreamConfig API). It would be nice if we could ask a running eventgate service what streams it was allowed to produce (and what streams it knows about).

We should add a /stream-configs route that will allow us to query eventgate for this info. This can be used to aide in automated ingestions of streams and to figure out what events we should produce canary events for.

See also T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events and T250844: MEP: canary events so we know events are flowing through pipeline

Event Timeline

Change 597341 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Create /stream-configs route to expose runtime stream configuration

https://gerrit.wikimedia.org/r/597341

Change 597341 merged by Ottomata:
[eventgate-wikimedia@master] Create /stream-configs route to expose runtime stream configuration

https://gerrit.wikimedia.org/r/597341

Milimetric moved this task from Incoming to Event Platform on the Analytics board.

ping @Ottomata

Whet is the path where stream configs are surfaced to?

/v1/stream-configs, e.g.

 curl -s https://intake-analytics.wikimedia.org/v1/stream-configs | jq .
{
  "eventgate-analytics-external.error.validation": {
    "topics": [
      "eventgate-analytics-external.error.validation"
    ],
    "schema_title": "error",
    "stream": "/^eventgate-.+\\.error(\\..+)?/"
  },
  "eventlogging_SearchSatisfaction": {
    "topics": [
      "eventlogging_SearchSatisfaction"
    ],
    "schema_title": "analytics/legacy/searchsatisfaction",
    "stream": "eventlogging_SearchSatisfaction"
  },
  "test.event": {
    "topics": [
      "test.event"
    ],
    "schema_title": "test/event",
    "stream": "test.event"
  }
}

HM! Which shows me that EventStreamConfig's topics inference is incorrect. Filing a bug. :)

so nice to see, this, great progress!