Name: WDQS Streaming Updater
Description: The WDQS Streaming Updater allows generating a stream of updates for WDQS. This service is a replacement for part of the current updater that is deployed on each WDQS server. It allows consolidating the common part of that update process instead duplicating that processing on each node and improves throughput by 5-10x.
Timeline: Q4 2021, hopefully May. Discussions have been ongoing for a few month, so while this timeline is short, it is not unexpected.
Point person: @RKemper as SRE, @dcausse and @Zbyszko as SWE, Search Platform as a team.
Technologies:
- Flink: a stateful stream processing framework (Java based)
- Java / JVM
- Kafka (both as a consumer and a producer)
- Running on k8s, with some direct dependencies (ability to publish config maps)
Request flow diagram:
Misc: