Page MenuHomePhabricator

tools-exec-1213 looks dead
Closed, ResolvedPublic

Description

Graphite load & process data null since 27th:

graphite.wmflabs.org.png (250×800 px, 41 KB)

http://tools.wmflabs.org/?status shows tools-exec-1213 Load: 0% Memory: 0% Free vmem: 12.1G

SSH connections fail:

12:07:50 0 ✓ zhuyifei1999@tools-bastion-02: ~$ ssh -vvv tools-exec-1213
OpenSSH_6.9p1 Ubuntu-2~trusty1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to tools-exec-1213 [10.68.17.252] port 22.
debug1: Connection established.
debug1: key_load_private_type: No such file or directory
debug1: key_load_private_cert: Permission denied
debug1: key_load_private_cert: Permission denied
debug1: key_load_private_cert: Permission denied
debug1: key_load_private_cert: Permission denied
debug1: key_load_private_type: Permission denied
debug1: key_load_private_type: Permission denied
debug1: key_load_private_type: Permission denied
debug1: key_load_private_type: Permission denied
debug1: key_load_cert: No such file or directory
debug1: key_load_cert: No such file or directory
debug1: key_load_cert: No such file or directory
debug1: key_load_cert: No such file or directory
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/zhuyifei1999/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9p1 Ubuntu-2~trusty1
ssh_exchange_identification: read: Connection reset by peer

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
Resolved chasemp

Event Timeline

zhuyifei1999 raised the priority of this task from to Needs Triage.
zhuyifei1999 updated the task description. (Show Details)
zhuyifei1999 added a project: Toolforge.
zhuyifei1999 subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

Cleaned up continuous jobs, but there are unfortunately also a few 'task' jobs running:

2558791 0.33570 radehc     tools.rezabo r     01/21/2016 22:17:04 task@tools MASTER
 2598902 0.33337 prb        tools.yifeib r     01/23/2016 00:17:09 task@tools MASTER
 2696179 0.32541 zumranew   tools.shuaib r     01/26/2016 17:15:07 task@tools MASTER
 2698283 0.32526 rdallvoy   tools.avicbo r     01/26/2016 19:00:33 task@tools MASTER

so I'm leaving the host up for now (although I'm not sure if these tasks are actually still running...)

The rescheduled jobs are

   5580 0.80000 vandalstat tools.cluest Rr    01/21/2016 18:43:27 continuous
   7971 0.41375 AnomieBOT- tools.anomie Rr    01/21/2016 18:35:08 continuous
 703867 0.53808 lrbot      tools.lrbot  Rr    01/21/2016 18:35:08 continuous
1915850 0.67351 toolhistor tools.admin  Rr    01/21/2016 18:35:08 continuous
2551855 0.33597 cluebot3   tools.cluebo r     01/21/2016 19:13:24 continuous

and I force-deleted

287082 0.42262 rmiw.w3    tools.yifeib dRr   01/21/2016 19:00:03 continuous
chasemp claimed this task.
chasemp subscribed.

unresponsive to salt so reboot