Page MenuHomePhabricator

Quarry struggling on the queue (2019-10-01)
Closed, ResolvedPublic

Description

https://quarry.wmflabs.org/query/runs/all is struggling to process queues for about a day (22 hours).

Event Timeline

revi created this task.Oct 1 2019, 9:58 AM
revi moved this task from Incoming to Radar on the User-revi board.
zhuyifei1999 closed this task as Resolved.Oct 1 2019, 11:56 AM
zhuyifei1999 claimed this task.
zhuyifei1999 added a subscriber: zhuyifei1999.

I ssh-ed in to both workers, no weird behavior. No logs either. It's as if the celery workers never received the tasks.

Ran sudo systemctl restart celery-quarry-worker.service on both of them, finished almost immediately, not even 'waiting for running tasks to finish'... so there is no running tasks. Celery broke?

In any case, a celery restart on both workers seemed to fix it.