List of steps to reproduce (step by step, including full links if applicable):
What happens?:
Query takes several minutes to execute (in the linked example, 493 seconds)
What should have happened instead?:
Query should be able to complete in fractions of a second (running that same query from a toolforge shell takes 0.001 seconds)
Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc.:
Per https://www.mediawiki.org/wiki/Phabricator/Project_management#Priority_levels: UBN! is reserved for isses meeting the guidelines at https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#Issues_that_hold_the_train.
Looks like a few long running processes were locking up nfs on one of the workers. Thus quarry worked about half the time (the half that it got the not loaded worker). Cleared the old jobs and restarted the worker process. So far things seem to be running fine. Will check again tomorrow. Please comment if you're not seeing the same.
Both workers look like they have low load. No hanging lsof processes. Queries seem to be returning at a reasonable speed.