Change Details

==== **User Story** > ==== As a platform engineer, I need to design, implement and deploy a Flink service that listens to existing event streams, consolidates them, enriches the event and outputs to Kafka ==== Timebox: - 3 weeks (including data modeling?) ==== The service must: [] Listen to the following existing streams: - mediawiki.page-create - mediawiki.page-delete - mediawiki.revision-create - any others? Undelete? [] Call MW API to get the wikitext for the article [] Format the input stream data and wikitext into the new topic format [] Output the formatted data to a new Kafka topic ==== Done is: [] Service deployed running on POC instance of Flink in YARN, producing to the kafka test cluster in eqiad. (no SLO’s) ==== Expected Spikes: [] Data modeling exercise for new consolidated stream [] Others? ==== Why are we doing this? - Simplify event stream consumption. Consumers can listen to a single stream that represent the state of a page rather than a page action (current design) - Adding content to streams to make them usable by consumers without having to enrich themselves

==== **User Story** > ==== As a platform engineer, I need to design, implement and deploy a Flink service that listens to existing event streams, consolidates them, enriches the event and outputs to Kafka ==== The service must: [] Call MW API to get the wikitext for the article [] Format the input stream data and wikitext into the new topic format [] Output the formatted data to a new Kafka topic ==== Expected Spikes: [] Data modeling exercise for new consolidated stream [] Others? ==== Why are we doing this? - Simplify event stream consumption. Consumers can listen to a single stream that represent the state of a page rather than a page action (current design) - Adding content to streams to make them usable by consumers without having to enrich themselves

==== **User Story** > ==== As a platform engineer, I need to design, implement and deploy a Flink service that listens to existing event streams, consolidates them, enriches the event and outputs to Kafka ==== Timebox: - 3 weeks (including data modeling?) ==== The service must: [] Listen to the following existing streams: - mediawiki.page-create - mediawiki.page-delete - mediawiki.revision-create - any others? Undelete? [] Call MW API to get the wikitext for the article [] Format the input stream data and wikitext into the new topic format [] Output the formatted data to a new Kafka topic ==== Done is: [] Service deployed running on POC instance of Flink in YARN, producing to the kafka test cluster in eqiad. (no SLO’s) ==== Expected Spikes: [] Data modeling exercise for new consolidated stream [] Others? ==== Why are we doing this? - Simplify event stream consumption. Consumers can listen to a single stream that represent the state of a page rather than a page action (current design) - Adding content to streams to make them usable by consumers without having to enrich themselves