A link for example for non-developers of Quarry: 27365.

Tried a very simple query, that usually runs for seconds, 72209, and it already runs for five minutes. The Recent Queries board shows that it happenes to all the queries in the last hour.

In fact, no query has finished in the last 8 hours.

Looks like the storage is full

quarry-nfs.svc.quarry.eqiad1.wikimedia.cloud:/srv/quarry/project  393G  373G     0 100% /mnt/nfs/labstore-secondary-project

I've cleaned up the storage. There were some very large query results in there. Looks working now, though yinz may have to restart the queries that got stuck while the storage was full.

Thanks a lot.

@rook: Is there some kind of alerting when storage reads a certain threshold?

Maybe you should kill the 33 still running queries.

In T334903#8788998, @IKhitron wrote:

Maybe you should kill the 33 still running queries.

Anyone? Nine querries are still running, and using resources. Thank you.

@IKhitron oh I wouldn't worry about it, I don't suspect that the queries are still actually running, this behavior is tracked in T278583
"running" isn't a reflection of a query actually doing any processing. It's just a label in the db, that usually relates to if the query is running or not.

I see. Thank you.