Today three celery-ores-worker services failed at around the same time on ores100[2,3,5]. From the logs the only thing that I can see is:
```
Aug 21 11:31:12 ores1002 celery-ores-worker[4251]: /srv/deployment/ores/deploy-cache/revs/d08fa628aacb82529dbb4be357b68dd55c15fdee/venv/lib/python3.5/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
Aug 21 11:31:12 ores1002 celery-ores-worker[4251]: 'See the migration notes for details: %s' % _MIGRATION_NOTES_URL
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]:
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -------------- celery@ores1002 v4.1.1 (latentcall)
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: ---- **** -----
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: --- * *** * -- Linux-4.9.0-9-amd64-x86_64-with-debian-9.9 2019-08-06 17:36:49
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -- * - **** ---
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- [config]
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- .> app: ores.scoring_systems.celery_queue:0x7fa11ed45278
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- .> transport: redis://:**@oresrdb.svc.eqiad.wmnet:6379//
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- .> results: redis://:**@oresrdb.svc.eqiad.wmnet:6379/
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - *** --- * --- .> concurrency: 90 (prefork)
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: --- ***** -----
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -------------- [queues]
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: .> celery exchange=celery(direct) key=celery
Aug 21 11:31:13 ores1002 celery-ores-worker[4251]:
```
The code path doesn't seem to lead to any shutdown, and the rest seems to be the log that celery emits when shutting down (it can be seen before in the logs).
Also we got alarmed by the systemd unit failed alert, not by any celery specific ones (not sure if we don't alarm on purpose or if we are missing some monitor).