Page MenuHomePhabricator

What is causing ORES celery workers to suddenly require more CPU?
Closed, ResolvedPublic

Description

This isn't a behavior I'm familiar with and I don't understand it yet, but the graphs show level CPU usage around 20%, then sudden spikes during our problem periods. Also noted by a diligent services node watcher:

mobrovac: awight: from experience, ores is prone to have such episodes, the celery workers have a high chance of using 100% of cpu whenever i look at top on scb nodes
mobrovac: usually with each of them using at least 3%, 4% of mem

Event Timeline

@akosiaris I'd like to get our celery logs routed to logstash, at INFO level. We could just pipe into a file too, per your judgment.

And yes let's send these logs to logstash!!!

Ottomata triaged this task as Medium priority.Jan 16 2018, 8:08 PM
Ladsgroup claimed this task.
Ladsgroup subscribed.

The logs are in logstash and we haven't seen celery high usage for a very long time now. I call this resolved.