Change Details

Today three celery-ores-worker services failed at around the same time on ores100[2,4,5]. From the logs the only thing that I can see is: ``` Aug 21 11:31:12 ores1002 celery-ores-worker[4251]: /srv/deployment/ores/deploy-cache/revs/d08fa628aacb82529dbb4be357b68dd55c15fdee/venv/lib/python3.5/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function Aug 21 11:31:12 ores1002 celery-ores-worker[4251]: 'See the migration notes for details: %s' % _MIGRATION_NOTES_URL Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -------------- celery@ores1002 v4.1.1 (latentcall) Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: ---- **** ----- Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: --- * *** * -- Linux-4.9.0-9-amd64-x86_64-with-debian-9.9 2019-08-06 17:36:49 Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -- * - **** --- Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- [config] Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- .> app: ores.scoring_systems.celery_queue:0x7fa11ed45278 Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- .> transport: redis://:**@oresrdb.svc.eqiad.wmnet:6379// Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - ** ---------- .> results: redis://:**@oresrdb.svc.eqiad.wmnet:6379/ Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: - *** --- * --- .> concurrency: 90 (prefork) Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker) Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: --- ***** ----- Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -------------- [queues] Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: .> celery exchange=celery(direct) key=celery Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: ``` The code path doesn't seem to lead to any shutdown, and the rest seems to be the log that celery emits when shutting down (it can be seen before in the logs). Also we got alarmed by the systemd unit failed alert, not by any celery specific ones (not sure if we don't alarm on purpose or if we are missing some monitor).

Today three celery-ores-worker services failed at around the same time on ores100[2,4,5]. From the logs the only thing that I can see is: ``` Aug 21 11:31:12 ores1005 celery-ores-worker[13281]: /srv/deployment/ores/deploy-cache/revs/d08fa628aacb82529dbb4be357b68dd55c15fdee/venv/lib/python3.5/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: h Aug 21 11:31:12 ores1005 celery-ores-worker[13281]: 'See the migration notes for details: %s' % _MIGRATION_NOTES_URL Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: -------------- celery@ores1005 v4.1.1 (latentcall) Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: ---- **** ----- Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: --- * *** * -- Linux-4.9.0-9-amd64-x86_64-with-debian-9.9 2019-08-06 17:36:46 Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: -- * - **** --- Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: - ** ---------- [config] Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: - ** ---------- .> app: ores.scoring_systems.celery_queue:0x7f6dab334ba8 Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: - ** ---------- .> transport: redis://:**@oresrdb.svc.eqiad.wmnet:6379// Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: - ** ---------- .> results: redis://:**@oresrdb.svc.eqiad.wmnet:6379/ Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: - *** --- * --- .> concurrency: 90 (prefork) Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker) Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: --- ***** ----- Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: -------------- [queues] Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: .> celery exchange=celery(direct) key=celery Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: Aug 21 11:31:16 ores1005 systemd[1]: celery-ores-worker.service: Main process exited, code=exited, status=1/FAILURE Aug 21 11:31:16 ores1005 systemd[1]: celery-ores-worker.service: Unit entered failed state. Aug 21 11:31:16 ores1005 systemd[1]: celery-ores-worker.service: Failed with result 'exit-code'. Aug 21 11:38:28 ores1005 systemd[1]: Started Celery workers. ``` The code path doesn't seem to lead to any shutdown, and the rest seems to be the log that celery emits when shutting down (it can be seen before in the logs). Also we got alarmed by the systemd unit failed alert, not by any celery specific ones (not sure if we don't alarm on purpose or if we are missing some monitor).

Today three celery-ores-worker services failed at around the same time on ores100[2,4,5]. From the logs the only thing that I can see is: ``` Aug 21 11:31:12 ores10025 celery-ores-worker[42513281]: /srv/deployment/ores/deploy-cache/revs/d08fa628aacb82529dbb4be357b68dd55c15fdee/venv/lib/python3.5/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function Aug 21 11:31:12 ores10025 celery-ores-worker[4251]:13281]: 'See the migration notes for details: %s' % _MIGRATION_NOTES_URL Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: Aug 21 11:31:13 ores1002 celery-ores-worker[4251]: -------------- celery@ores1002 v4.1.1 (latentcall)4 ores1005 celery-ores-worker[13281]: Aug 21 11:31:134 ores10025 celery-ores-worker[4251]:13281]: ---- **** ----------- celery@ores1005 v4.1.1 (latentcall) Aug 21 11:31:134 ores10025 celery-ores-worker[42513281]: --- *- *** * -- Linux-4.9.0-9-amd64-x86_64-with-debian-9.9 2019-08-06 17:36:49* ----- Aug 21 11:31:134 ores10025 celery-ores-worker[42513281]: --- * - **** ---*** * -- Linux-4.9.0-9-amd64-x86_64-with-debian-9.9 2019-08-06 17:36:46 Aug 21 11:31:134 ores10025 celery-ores-worker[42513281]: - ** ---------- [config]- * - **** --- Aug 21 11:31:134 ores10025 celery-ores-worker[42513281]: - ** ---------- .> app: ores.scoring_systems.celery_queue:0x7fa11ed45278[config] Aug 21 11:31:134 ores10025 celery-ores-worker[4251]:13281]: - ** ---------- .> transport: redis://:**@oresrdb.svc.eqiad.wmnet:6379//app: ores.scoring_systems.celery_queue:0x7f6dab334ba8 Aug 21 11:31:134 ores10025 celery-ores-worker[4251]:13281]: - ** ---------- .> results: transport: redis://:**@oresrdb.svc.eqiad.wmnet:6379// Aug 21 11:31:134 ores10025 celery-ores-worker[42513281]: - *** --- * ---------- .> concurrency: 90 (prefork)results: redis://:**@oresrdb.svc.eqiad.wmnet:6379/ Aug 21 11:31:134 ores10025 celery-ores-worker[42513281]: -- ****- *** ---- .> task events: OFF (enable -E to monitor tasks in this worker--- * --- .> concurrency: 90 (prefork) Aug 21 11:31:134 ores10025 celery-ores-worker[42513281]: --- ******* ----- .> task events: OFF (enable -E to monitor tasks in this worker) Aug 21 11:31:134 ores10025 celery-ores-worker[4251]: 13281]: ---------- ***** ---- [queues]- Aug 21 11:31:134 ores10025 celery-ores-worker[4251]: .> celery exchange=celery(direct) key=celery13281]: -------------- [queues] Aug 21 11:31:13 ores1002 celery-ores-worker[4251]:4 ores1005 celery-ores-worker[13281]: .> celery exchange=celery(direct) key=celery Aug 21 11:31:14 ores1005 celery-ores-worker[13281]: Aug 21 11:31:16 ores1005 systemd[1]: celery-ores-worker.service: Main process exited, code=exited, status=1/FAILURE Aug 21 11:31:16 ores1005 systemd[1]: celery-ores-worker.service: Unit entered failed state. Aug 21 11:31:16 ores1005 systemd[1]: celery-ores-worker.service: Failed with result 'exit-code'. Aug 21 11:38:28 ores1005 systemd[1]: Started Celery workers. ``` The code path doesn't seem to lead to any shutdown, and the rest seems to be the log that celery emits when shutting down (it can be seen before in the logs). Also we got alarmed by the systemd unit failed alert, not by any celery specific ones (not sure if we don't alarm on purpose or if we are missing some monitor).