Page MenuHomePhabricator

Weigh Kubernetes vs. streaming architecture for ORES "2.0"
Closed, DeclinedPublic

Description

We've started down two alternative reimplementations of ORES, and I suspect that they might be mutually exclusive. One is to keep ORES as a standalone service, but to implement worker nodes for k8 deployment. The other is to replace ORES with a pure cache-backed service, calculating new scores in a streaming paradigm.

It's possible that backfilling old scores could be done by k8 nodes, making the plans compatible, but I think there may be better options such as distributing the Python workload over the Hadoop cluster and rewriting other features such as our training.

We don't have spare resources for contradictory implementations, so should decide between these approaches soon, or find a way to reconcile.

Event Timeline

awight created this task.Feb 6 2019, 4:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 6 2019, 4:22 PM
awight updated the task description. (Show Details)Feb 6 2019, 4:22 PM
Halfak closed this task as Declined.Apr 2 2019, 9:34 PM
Halfak added a subscriber: Halfak.

Declining because we're following the rest of the org to K8s. We'll be considering streaming architecture more later. K8s and streaming are orthogonal.