vivian-rook opened https://github.com/toolforge/quarry/pull/38

I'm guessing this issue is from the threads being in separate pods. The attached PR removes the feature which may be the way to go with it.

I don't think that's the issue. We persist the db process id in the query_run table, so even a different pod is able to execute KILL <id> on the db to get the query to stop.

The issue I suspect is that *.analytics.db.svc.eqiad.wmflabs are LB endpoints behind which there could be multiple replicas (@taavi - would you be able to confirm if this is the case?). The KILL command wouldn't work if received by a different replica than the one on which query is running. This is also the reason the automated kills after the 30-minute timeout are unreliable nowadays.

In T362213#9705787, @SD0001 wrote:

The issue I suspect is that *.analytics.db.svc.eqiad.wmflabs are LB endpoints behind which there could be multiple replicas (@taavi - would you be able to confirm if this is the case?).

No, not usually. (The long answer is that there's generally one analytics and one web replica per section, and while maintenance sometimes means that we send all traffic to a single server, it should not lead to a situation where a single endpoint is routed to multiple replicas at the same time.)