Page MenuHomePhabricator

Tool Labs grid foobar
Closed, ResolvedPublic

Description

Several of my tools sent me mail about cronjob trouble. So I tried to list running jobs in the shell:

tools.quickstatements@tools-bastion-03:~$ qstat
error: commlib error: got select error (Connection refused)
error: unable to send message to qmaster using port 6444 on host "tools-grid-master.tools.eqiad.wmflabs": got send error

Please fix!

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
chasemp added subscribers: Andrew, chasemp.

thanks @Magnus, @Andrew and I are actively fighting something overwhelming the grid. We are still not sure what the deal is but had to restart the master process.

https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL

chasemp claimed this task.

sorted out in T161951