Hmm. Is the goal trying to find when a worker gets SIGKILL-ed? Celery does
internally detect when a worker dies, as per the logs, but I did not figure
out how to hook it so that it would report to the db.

At least one occurrence of this error can be reproduced with a simple query like "Select page_title, page_title from page where page_id = 1", see T265155 or look at https://quarry.wmflabs.org/query/53652

In T278583#6951002, @zhuyifei1999 wrote:

Hmm. Is the goal trying to find when a worker gets SIGKILL-ed? Celery does
internally detect when a worker dies, as per the logs, but I did not figure
out how to hook it so that it would report to the db.

When it gets SIGKILL-ed or maybe also if other things happen...but mostly when it gets SIGKILL-ed because that's what I see happening regularly. If there's some kind of hook to place in a try: except: raise that'd be slick in general as well because anything that does cause exception also causes the confusing state just like SIGKILL does.

Thanks @Wurgl, that reproducible example should help.

Overall, this is a bug that has it's own workboard column, so it seems like a place to spend some time if it has been confusing people for that many years.

Workers sometimes die unexpectedly, and leave the db in the state described "Running". One method to clean this up may be to inspect the running processes on the quarry database. As a dead worker will not have a running process, Any process that is in the running state and is older than the oldest process in quarry can be updated to "failed" from "running". I believe this will clean out any not really running processes that are older than an hour.