As a WME Engineer working on releasing a new product with realtime capabilities i'd like to do a full review of our current kafka consumer handler logic in order to accommodate a new product.

Requirements

• As a different product, Wikidata should have it's own limits, in terms of connections and rate limiting.
• As a different product, Wikidata should be possible to enable to different groups regardless of base product.
• Wikidata will have it's own model/schema therefore all consumers must be backwards compatible to allow schema evolutions
• As a separate product, metrics, traces and logging should be clearly identifiable for it's scope
• Filtering and Fields stripping should be traceable like any other event specific operations per event ( 10% sample, configurable for more )
• All logs and tracing should have a common and related tracing identifier. so we can correlate traces and logs.
• For sake of deprecation and operational effort, offset tracking must not be available for wikidata

Developer Notes:

Requirements should be look at by priority in terms of effort. Not everything needs to be in first release ... but a path forward should be considered.\

Acceptance Criteria

• RFC documenting changes proposed
• Discussion with team and agreement on next steps
• All tickets created for first release and after, so we can groom backlog.