The mediawiki.page_html_feature_counts_change.v1 stream contains very valuable data about what has changed in an edit to a wikipedia page.
We should expose this stream publicly via the EventStreams API so it is viewable and consumable outside of WMF production.
To to this, we need to
- produce the stream to Kafka main cluster
- modify EventStreams config to allowlist this stream.
This data is just as small as other page_change based streams on Kafka main, so there should be no worry about producing data there.
Explicit steps:
- Make the flink app produce to kafka main-eqiad cluster, edit here, deploy changes.
- Add to EventStreams allowed_streams. Edit here, deploy changes.
Deploying the flink app with these changes will stop writing to kafka jumbo and start writing to kafka main. eqiad.* prefixed topics are all automatically mirrored to Kafka jumbo, so MirrorMaker will start producing these messages to the same existing topic in Kafka jumbo. Ingestion in to Hive will continue to work with no changes.
So! I think this is a very simple config change!