Page MenuHomePhabricator

Query runs over 5 hours without being killed
Closed, ResolvedPublic

Description

Could you fix this:


quickly?

It's query 6990: https://quarry.wmflabs.org/query/6990

Event Timeline

Dvorapa created this task.Jul 1 2016, 2:07 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 1 2016, 2:07 PM
tom29739 updated the task description. (Show Details)Jul 1 2016, 2:11 PM
tom29739 added a subscriber: tom29739.
jcrespo added a subscriber: jcrespo.EditedJul 1 2016, 2:23 PM

Query limit is also enforced at server side, do not worry about this having any kind of impact on our infrastructure. It would be interesting as a report of the error/mistake/race condition/bug over it being shown as in running state (when in reality it is not).

Dvorapa added a comment.EditedJul 1 2016, 3:16 PM

@jcrespo But I wait more than 2 hours now and this query is completed, but the query above #6992 is now running more than 1 hour.

The bug you mean is not the same as this, but you can find it in T137517

Dvorapa added a comment.EditedJul 1 2016, 4:12 PM

Just one query completed in last 2 hours:

I just restarted the runner, hopefully that'll make things better.

Looks like working, thank you

I ran:

MariaDB [quarry]> UPDATE query join query_revision on query.latest_rev_id = query_revision.id  join query_run on latest_run_id = query_run.id SET status=1 where (status = 2 or status = 1) and query_run.timestamp <= DATE_ADD(NOW(), INTERVAL -1 HOUR);

Which clears out the status of the old stale queries for now. I need to actually spend time on fixing the underlying cause at some point...

Is this still an issue? I haven't seen this issue for a while.

Mess added a subscriber: Mess.Jun 26 2017, 5:02 PM

Is this still an issue? I haven't seen this issue for a while.

Yes, it is indeed @Dvorapa. Check out my recent report here: https://phabricator.wikimedia.org/T168803 (but you can verify it also having a look on the recent queries list, where there are a lot of very old queries still queued or busy). Unfortunately, Yuvipanda gave up maintaining the tool some days ago, so I don't know who can take care of it.

@Mess Recent queries list shows queued but the query is completed actually, you can see its real status if you click on some. The wrong status is tracked by T137517. But I think this issue should be closed, because in last year I haven't seen any query running more than 30 minutes.

Mess added a comment.Jun 26 2017, 9:10 PM

@Dvorapa It's not true: I've clicked on this query and this other one and their status is actually "queued" (and they were launched several hours ago); moreover, queries like this or this one are still running and won't stop. Meanwhile, my queries for orphaned articles in it.wiki took around 15-20 minutes until some days ago, but now they run out and are killed after 30 minutes.

@Mess OK then. I experience same slow down as you describe. Sometimes even not very complicated queries take too much time.

Mentioned in SAL (#wikimedia-cloud) [2018-03-01T00:37:23Z] <zhuyifei1999_> UPDATE query join query_revision on query.latest_rev_id = query_revision.id join query_run on latest_run_id = query_run.id SET status=1 where (status = 2 or status = 1) and query_run.timestamp <= DATE_ADD(NOW(), INTERVAL -1 HOUR); 251 rows affected (1.81 sec) T139162 T172086 T188564

Restricted Application added a subscriber: Zoranzoki21. · View Herald TranscriptMar 1 2018, 12:37 AM
Dvorapa closed this task as Resolved.Jul 29 2018, 8:38 AM

I have not seen this for a long time