Page MenuHomePhabricator

Wikidata Phase 1 - Page Change eventstream
Closed, ResolvedPublic

Description

As a WME Engineer i want meet wikidata product requirements by first retrieving to pagechange wikidata events from eventstream,

In order to do so, we can either build a new service or adapt the current eventstream listener to get the data. Current service already listens to pagechange and drops wikidata events.

Eventstream listener uses an command pattern to execute operations by message type and consequently output to different topics.

In the wikidata realm we want to follow the same approach, new service or adapting current, to allow wikidata events to be propagated. Unlike wikipedia events wikidata events should not output to different topics. The messages should output to a single topic with 50 partitions, in order for us to allow potential consumers to catch up and scale up enough to follow up with realtime traffic. We estimating a max of 60 events per second to be supported as peak.

Since we creating the kafka topic, we should also create the realtime topic with 50 partitions and a retention of TBD days.

Acceptance Criteria

  • Created eventstream wikidata listener output topic in dev
  • Created wikidata realtime topic in dev
  • Eventstream Listener supports wikidata in dev. team decision is to modify the current event listener to write currently skipped wikidata events to intercept them and then write to the new topic created above.
  • Metrics available in prometheus dev.
  • Graphana dashboard on dev