Page MenuHomePhabricator

Coordinate with Platform Engineering / Data Value Stream Team about a rework of the Search Update Pipeline
Closed, ResolvedPublic

Description

See parent task and https://docs.google.com/document/d/17tY05WoaT_BloTzaIncR939k3hvhcVQ-E-8DBjo284E/edit# for context.

The rework of the Search Update Pipeline is planned to be implemented on top of Flink. The Data Value Stream seem to be owning this technical component, so collaboration will be required. The current plan isn't stable yet, but we envision the following needs from the Search Platform team to the Data Value Stream team:

  • collaboration on defining the exact solution, taking into account contraints from both teams
  • deployment and maintenance of a high availability Flink cluster
  • standardized stream of Media wiki mutations (create / update / delete / ...)

AC:

  • we had a kickoff meeting

Event Timeline

@JArguello-WMF It should get tagged for Event-Platform and @lbowmaker is I think subscribed. I'll add the tags.

Kick off meeting done. Further collaboration is expected to happen on specific subtasks of T317045 or as discussion on https://docs.google.com/document/d/17tY05WoaT_BloTzaIncR939k3hvhcVQ-E-8DBjo284E/edit