Tracked down this query:
SELECT * FROM categorylinks
MariaDB [quarry]> select * from query where id = 20534; +-------+---------+-------+---------------+---------------------+-----------+-----------+---------------------------+ | id | user_id | title | latest_rev_id | last_touched | parent_id | published | description | +-------+---------+-------+---------------+---------------------+-----------+-----------+---------------------------+ | 20534 | 2021 | NULL | 197241 | 2017-07-27 12:21:21 | NULL | 0 | Select all category links | +-------+---------+-------+---------------+---------------------+-----------+-----------+---------------------------+ 1 row in set (0.00 sec) MariaDB [quarry]> select * from query_run where query_rev_id = 197241; +--------+--------------+--------+---------------------+--------------------------------------+------------+ | id | query_rev_id | status | timestamp | task_id | extra_info | +--------+--------------+--------+---------------------+--------------------------------------+------------+ | 193235 | 197241 | 2 | 2017-07-27 12:21:21 | 20f5aad9-64c5-4f5b-8360-5fcc4e4de5c6 | NULL | +--------+--------------+--------+---------------------+--------------------------------------+------------+ 1 row in set (0.00 sec)
zhuyifei1999@quarry-runner-01:~$ zcat /var/log/syslog*.gz | grep 20f5aad9-64c5-4f5b-8360-5fcc4e4de5c6 -A 5 Jul 27 12:34:05 quarry-runner-01 celery-quarry-worker[591]: [2017-07-27 12:34:05,006: ERROR/MainProcess] Task worker.run_query[20f5aad9-64c5-4f5b-8360-5fcc4e4de5c6] raised unexpected: Terminated(9,) Jul 27 12:34:05 quarry-runner-01 celery-quarry-worker[591]: Traceback (most recent call last): Jul 27 12:34:05 quarry-runner-01 celery-quarry-worker[591]: File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 1673, in _set_terminated Jul 27 12:34:05 quarry-runner-01 celery-quarry-worker[591]: raise Terminated(-(signum or 0)) Jul 27 12:34:05 quarry-runner-01 celery-quarry-worker[591]: Terminated: 9 Jul 27 12:35:01 quarry-runner-01 CRON[21232]: (prometheus) CMD (/usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom)
It is probably an OOM due to the excessive data size. Unfortunately SIGKILL cannot be caught by the inner worker process and therefore kept a 'Running' state.