Page MenuHomePhabricator
Create Task
Maniphest T156168

Tool labs crontab host tools-cron-01 cannot be ssh-ed into within a service user
Closed, ResolvedPublic
Actions

Description

Tool labs crontabs are stored/executed on tools-cron-01, but it is currently inaccessible within a service user:

tools.yifeibot@tools-bastion-02:~$ crontab -l
Connection closed by 10.68.23.89

Debug log while sshing manually:

P4798 SSH into tools-cron-01 debug log on client
1tools.yifeibot@tools-bastion-02:~$ crontab -l
2Connection closed by 10.68.23.89
3tools.yifeibot@tools-bastion-02:~$ ssh -vvv tools-cron-01
4OpenSSH_6.9p1 Ubuntu-2~trusty1, OpenSSL 1.0.1f 6 Jan 2014
5debug1: Reading configuration data /etc/ssh/ssh_config
6debug1: /etc/ssh/ssh_config line 20: Applying options for *
7debug2: ssh_connect: needpriv 0
8debug1: Connecting to tools-cron-01 [10.68.23.89] port 22.
9debug1: Connection established.
10debug1: key_load_private_type: No such file or directory
11debug1: key_load_private_cert: Permission denied
12debug1: key_load_private_cert: Permission denied
13debug1: key_load_private_cert: Permission denied
14debug1: key_load_private_cert: Permission denied
15debug1: key_load_private_type: Permission denied
16debug1: key_load_private_type: Permission denied
17debug1: key_load_private_type: Permission denied
18debug1: key_load_private_type: Permission denied
19debug1: key_load_cert: No such file or directory
20debug1: key_load_cert: No such file or directory
21debug1: key_load_cert: No such file or directory
22debug1: key_load_cert: No such file or directory
23debug1: key_load_public: No such file or directory
24debug1: identity file /data/project/yifeibot/.ssh/id_rsa type -1
25debug1: key_load_public: No such file or directory
26debug1: identity file /data/project/yifeibot/.ssh/id_rsa-cert type -1
27debug1: key_load_public: No such file or directory
28debug1: identity file /data/project/yifeibot/.ssh/id_dsa type -1
29debug1: key_load_public: No such file or directory
30debug1: identity file /data/project/yifeibot/.ssh/id_dsa-cert type -1
31debug1: key_load_public: No such file or directory
32debug1: identity file /data/project/yifeibot/.ssh/id_ecdsa type -1
33debug1: key_load_public: No such file or directory
34debug1: identity file /data/project/yifeibot/.ssh/id_ecdsa-cert type -1
35debug1: key_load_public: No such file or directory
36debug1: identity file /data/project/yifeibot/.ssh/id_ed25519 type -1
37debug1: key_load_public: No such file or directory
38debug1: identity file /data/project/yifeibot/.ssh/id_ed25519-cert type -1
39debug1: Enabling compatibility mode for protocol 2.0
40debug1: Local version string SSH-2.0-OpenSSH_6.9p1 Ubuntu-2~trusty1
41debug1: Remote protocol version 2.0, remote software version OpenSSH_6.9p1 Ubuntu-2~trusty1
42debug1: match: OpenSSH_6.9p1 Ubuntu-2~trusty1 pat OpenSSH* compat 0x04000000
43debug2: fd 3 setting O_NONBLOCK
44debug1: Authenticating to tools-cron-01:22 as 'tools.yifeibot'
45debug3: hostkeys_foreach: reading file "/data/project/yifeibot/.ssh/known_hosts"
46debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
47debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:11
48debug3: record_hostkey: found key type ECDSA in file /etc/ssh/ssh_known_hosts:12
49debug3: load_hostkeys: loaded 2 keys from tools-cron-01
50debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa
51debug1: SSH2_MSG_KEXINIT sent
52debug1: SSH2_MSG_KEXINIT received
53debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
54debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-ed25519-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-dss
55debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
56debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
57debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
58debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
59debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
60debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
61debug2: kex_parse_kexinit:
62debug2: kex_parse_kexinit:
63debug2: kex_parse_kexinit: first_kex_follows 0
64debug2: kex_parse_kexinit: reserved 0
65debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
66debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
67debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
68debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
69debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
70debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
71debug2: kex_parse_kexinit: none,zlib@openssh.com
72debug2: kex_parse_kexinit: none,zlib@openssh.com
73debug2: kex_parse_kexinit:
74debug2: kex_parse_kexinit:
75debug2: kex_parse_kexinit: first_kex_follows 0
76debug2: kex_parse_kexinit: reserved 0
77debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> none
78debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> none
79debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
80debug1: Server host key: ecdsa-sha2-nistp256 SHA256:gOfWfj+BwGF5QxfNzI27E5LsiXAJzRekrKClX3D6gZc
81debug3: hostkeys_foreach: reading file "/data/project/yifeibot/.ssh/known_hosts"
82debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
83debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:11
84debug3: record_hostkey: found key type ECDSA in file /etc/ssh/ssh_known_hosts:12
85debug3: load_hostkeys: loaded 2 keys from tools-cron-01
86debug3: hostkeys_foreach: reading file "/data/project/yifeibot/.ssh/known_hosts"
87debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
88debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:11
89debug3: record_hostkey: found key type ECDSA in file /etc/ssh/ssh_known_hosts:12
90debug3: load_hostkeys: loaded 2 keys from 10.68.23.89
91debug1: Host 'tools-cron-01' is known and matches the ECDSA host key.
92debug1: Found key in /etc/ssh/ssh_known_hosts:12
93debug2: set_newkeys: mode 1
94debug1: SSH2_MSG_NEWKEYS sent
95debug1: expecting SSH2_MSG_NEWKEYS
96debug2: set_newkeys: mode 0
97debug1: SSH2_MSG_NEWKEYS received
98debug1: Roaming not allowed by server
99debug1: SSH2_MSG_SERVICE_REQUEST sent
100debug2: service_accept: ssh-userauth
101debug1: SSH2_MSG_SERVICE_ACCEPT received
102debug2: key: /data/project/yifeibot/.ssh/id_rsa ((nil)),
103debug2: key: /data/project/yifeibot/.ssh/id_dsa ((nil)),
104debug2: key: /data/project/yifeibot/.ssh/id_ecdsa ((nil)),
105debug2: key: /data/project/yifeibot/.ssh/id_ed25519 ((nil)),
106debug1: Authentications that can continue: publickey,hostbased
107debug3: start over, passed a different list publickey,hostbased
108debug3: preferred gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
109debug3: authmethod_lookup hostbased
110debug3: remaining preferred: publickey,keyboard-interactive,password
111debug3: authmethod_is_enabled hostbased
112debug1: Next authentication method: hostbased
113debug3: userauth_hostbased: trying key type *
114debug1: userauth_hostbased: trying hostkey ecdsa-sha2-nistp256 SHA256:OfgR6GTw8ObBQ1LbS+6NBVik1eEXrpSUvRkKOueUnQc
115debug2: userauth_hostbased: chost tools-bastion-02.tools.eqiad.wmflabs.
116debug3: ssh_msg_send: type 2
117debug1: permanently_drop_suid: 51201
118debug3: ssh_msg_recv entering
119debug3: ssh_keysign: [child] pid=5015, exec /usr/lib/openssh/ssh-keysign
120debug2: we sent a hostbased packet, wait for reply
121Connection closed by 10.68.23.89

SSHing from user account (i.e. before using become) works. Crontabs are intact.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
tool: convert HBA source host mechanism to staticoperations/puppetproduction+23 -58
tools: specify ipaddress_eth0 for HBAoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

zhuyifei1999 created this task.Jan 24 2017, 7:47 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJan 24 2017, 7:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Framawiki subscribed.Jan 24 2017, 7:56 PM
gerritbot subscribed.Jan 24 2017, 8:39 PM

Change 333978 had a related patch set uploaded (by BryanDavis):
tools: specify ipaddress_eth0 for HBA

https://gerrit.wikimedia.org/r/333978

gerritbot added a project: Patch-For-Review.Jan 24 2017, 8:39 PM

Change 333978 merged by Rush:
tools: specify ipaddress_eth0 for HBA

https://gerrit.wikimedia.org/r/333978

bd808 mentioned this in T156174: Rewrite /usr/local/bin/crontab in python; fix bugs.Jan 24 2017, 8:43 PM
• chasemp subscribed.Jan 24 2017, 8:59 PM

seems to work now from bastion-03 to tools-cron-01 using crontab -e as a test tool. Thanks for the report @zhuyifei1999 this was a weird one to track down :)

• chasemp closed this task as Resolved.Jan 24 2017, 8:59 PM
• chasemp claimed this task.
MusikAnimal added a comment.Jan 24 2017, 9:52 PM

Thanks for the quick fix!

gerritbot added a comment.Jan 25 2017, 9:51 PM

Change 334203 had a related patch set uploaded (by BryanDavis):
tool: convert HBA source host mechanism to static

https://gerrit.wikimedia.org/r/334203

gerritbot added a comment.Aug 12 2017, 5:51 PM

Change 334203 abandoned by Rush:
tool: convert HBA source host mechanism to static

Reason:
come back to this another time,

https://gerrit.wikimedia.org/r/334203

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits