Page MenuHomePhabricator

[Discuss] ORES without celery
Open, NormalPublic

Description

Why do we run ORES with celery?

Celery allows us to separate IO requests from CPU parallelization. This is most useful for batch processing. We are able to request data needed for scoring in batch and then split the data into celery "tasks" for the CPU intensive work. 20-25% of our scores are delivered via batch requests[1] and a batch request is generally ~3x faster than requesting single scores. Our batch request use-case to increase in frequency over time.

If we could allow people to send 3x as many single-score requests, then the tradeoff would cancel out on our end. I don't think we could triple our workers if we dropped celery but we could potentially increase them by 20% or so with the memory we'd make available.

For what it's worth, we can run already run ORES in "single-threaded" mode (each UWSGI worker does both IO and CPU one task at a time) so this should be easy to test.

Why might we want to run ORES without celery?

In T122676#4974717, @Joe suggested that dropping celery would make ORES "behave more like the rest of our services. Uniformity is important."

What costs and benefits exist for running ORES without celery

This is what we mean to flesh out in this task.

This task is done when we have laid out a proposal for and discussed the considerations of deploying ORES without celery.

Event Timeline

Halfak created this task.Feb 22 2019, 5:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 22 2019, 5:06 PM

Technically there is no need to drop separation of IO requests from CPU parallelization. EventBus is based on kafka and at the end kafka is the same as celery (it has concept of producers and consumers). I only read about kafak and never worked with it so take what I said with grain of salt.

Halfak added a comment.EditedFeb 25 2019, 11:29 PM

Aaron misunderstands @Ladsgroup and then deletes his comment when he realizes his mistake

Oh! I misread. I agree with your point about not needing to drop it. I think the idea of dropping it comes from @Joe's proposal that we make ORES look like other services.

If you'd like to propose a complete engineering of ORES to work on top of Kafka, I'd really like to see that, but maybe it's out of scope for this task.

jbond triaged this task as Normal priority.Mar 4 2019, 7:44 PM
Halfak moved this task from Ideas to Monitor on the Scoring-platform-team board.Tue, Apr 2, 9:40 PM