There is a complex workflow that powers the Kafka stream mediawiki.revision-score, used in various part of our infrastructure:
- An editor makes a change to a wiki page, Mediawiki serves it and creates a mediawiki.revision-create event.
- The event is sent to EventGate, that validates it against a schema and saves it in a Kafka topic (called mediawiki.revision-create).
- Changeprop is instructed to read every event in the Kafka revision-create topic, calling ORES to get a score (with the /precache URI). The score is then sent to EventGate as mediawiki.revision-score event.
- EventGate validates it, and inserts it into the related Kafka revision-score topic.
Multiple consumers of the mediawiki.revision-create topic use the data for various purposes:
- Eventstreams exposes the stream publicly via https://stream.wikimedia.org/v2/ui/#/?streams=mediawiki.revision-score
- Data Engineering pull the topic hourly to ingest the data in Hadoop HDFS, exposing it via Hive (event database, table mediawiki.revision-score).
The main problem with the above architecture is that ChangeProp has a special hack to call ORES and create the revision-score event, meanwhile it should be ORES that should send it (leaving ChangeProp more ignorant and simple).
Since we are building Lift Wing, we should probably add the config/code that it is needed to send the revision-score event when requested. Some thoughts:
- The first thing that comes to mind is Knative eventing, but we are not currently deploying it via helm and our version of Knative is very old (since more recent ones require an higher version of K8s). So I'd avoid using it for the moment.
- We could add a simple Python function to send an HTTP POST to EventGate, it should be relatively easy to do.
- We can implement the above with a flag passed as parameter, but then we shouldn't expose it to API-Gateway (since we want only some clients to be able to trigger this). Changeprop will probably call the inference.discovery.wmnet endpoint, so it could be the only one allowed to use the feature flag.
- To avoid duplicated events, we should probably use (at least for the moment) a new EventGate schema, not mediawiki.revision-score. We can come up with a new name and talk with Data Engineering about it. Then we could experiment with it, maybe instructing Changeprop to call Lift Wing once the MVP is ready. We'd have a nice test pipeline to see how things go, and where fixes are needed.