(This is a parent/placeholder ticket for Q4 goals linking.)
https://meta.wikimedia.org/wiki/Research:MediaWiki_events:_a_generalized_public_event_datasource is a proposal that expands on the feature set currently available from RCStream. We would like to generalize this beyond just mediawiki events, and build a service that can make arbitrary event streams of JSON events available for public consumption.
A brainstorm meeting about this was held on March 15 2016. Notes from the meeting are here: https://etherpad.wikimedia.org/p/PublicEventBus
- Build a service that exposes configured Kafka topics via websockets. Offset/timestamp historical consumption and field filtering TBD. This should at least be feature compatible with current RCStream (e.g. wiki filtering).
- Expose public events currently available in Kafka via this service.
- Produce recent changes events to Kafka (possibly via EventBus service, but maybe not).
- Serve recent changes events from this service using backwards compatible API as RCStream.
- deprecate RCStream python/redis based service.