Page MenuHomePhabricator
Create Task
Maniphest T156580

Newly created instance requires password to ssh in
Closed, ResolvedPublic
Actions

Description

Yesterday I created the ubuntu-14.04-trusty instance 6dce7924-ed1f-463b-b343-31d5db974203 (nemobis; an instance with the same name existed in the past as well). I used Special:NovaInstance as usual.

ssh nemobis@nemobis.eqiad.wmflabs doesn't work: it asks for password. According to https://wikitech.wikimedia.org/wiki/Help:Access#Into_your-instance this might have to do with puppet not having been run yet, but it's been a day already.

P4827 Labs ssh password
1$ ssh -vi /home/federico/.ssh/id_rsa nemobis@nemobis.eqiad.wmflabs
2OpenSSH_7.4p1, OpenSSL 1.0.2j-fips 26 Sep 2016
3debug1: Reading configuration data /home/federico/.ssh/config
4debug1: /home/federico/.ssh/config line 32: Applying options for *.eqiad.wmflabs
5debug1: /home/federico/.ssh/config line 35: Applying options for *.wmflabs
6debug1: Reading configuration data /etc/ssh/ssh_config
7debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
8debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 2: include /etc/crypto-policies/back-ends/openssh.txt matched no files
9debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
10debug1: Executing proxy command: exec ssh -a -W nemobis.eqiad.wmflabs:22 bastion1.eqiad.wmflabs
11debug1: permanently_drop_suid: 1000
12debug1: identity file /home/federico/.ssh/id_rsa type 1
13debug1: key_load_public: No such file or directory
14debug1: identity file /home/federico/.ssh/id_rsa-cert type -1
15debug1: Enabling compatibility mode for protocol 2.0
16debug1: Local version string SSH-2.0-OpenSSH_7.4
17debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
18debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000
19debug1: Authenticating to nemobis.eqiad.wmflabs:22 as 'nemobis'
20debug1: SSH2_MSG_KEXINIT sent
21debug1: SSH2_MSG_KEXINIT received
22debug1: kex: algorithm: curve25519-sha256@libssh.org
23debug1: kex: host key algorithm: ecdsa-sha2-nistp256
24debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
25debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
26debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
27debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
28debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
29debug1: Server host key: ecdsa-sha2-nistp256 SHA256:UumyKsLDbRetAPCa/JkSEqTiYRVJxF0CorO5wZ91ggY
30debug1: Host 'nemobis.eqiad.wmflabs' is known and matches the ECDSA host key.
31debug1: Found key in /home/federico/.ssh/known_hosts:53
32debug1: rekey after 134217728 blocks
33debug1: SSH2_MSG_NEWKEYS sent
34debug1: expecting SSH2_MSG_NEWKEYS
35debug1: SSH2_MSG_NEWKEYS received
36debug1: rekey after 134217728 blocks
37debug1: Skipping ssh-dss key /home/federico/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes
38debug1: SSH2_MSG_SERVICE_ACCEPT received
39debug1: Authentications that can continue: publickey,keyboard-interactive
40debug1: Next authentication method: publickey
41debug1: Offering RSA public key: /home/federico/.ssh/id_rsa
42debug1: Authentications that can continue: publickey,keyboard-interactive
43debug1: Next authentication method: keyboard-interactive

My keys and ssh are set up correctly. Compare what happens with another instance:

P4830 Successful ssh into labs instance via bastion
1[federico@a88x ~]$ ssh -v nemobis@dumps-1.eqiad.wmflabs
2OpenSSH_7.4p1, OpenSSL 1.0.2j-fips 26 Sep 2016
3debug1: Reading configuration data /home/federico/.ssh/config
4debug1: /home/federico/.ssh/config line 32: Applying options for *.eqiad.wmflabs
5debug1: /home/federico/.ssh/config line 35: Applying options for *.wmflabs
6debug1: Reading configuration data /etc/ssh/ssh_config
7debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
8debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 2: include /etc/crypto-policies/back-ends/openssh.txt matched no files
9debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
10debug1: Executing proxy command: exec ssh -a -W dumps-1.eqiad.wmflabs:22 bastion1.eqiad.wmflabs
11debug1: permanently_drop_suid: 1000
12debug1: identity file /home/federico/.ssh/id_rsa type 1
13debug1: key_load_public: No such file or directory
14debug1: identity file /home/federico/.ssh/id_rsa-cert type -1
15debug1: identity file /home/federico/.ssh/id_dsa type 2
16debug1: key_load_public: No such file or directory
17debug1: identity file /home/federico/.ssh/id_dsa-cert type -1
18debug1: key_load_public: No such file or directory
19debug1: identity file /home/federico/.ssh/id_ecdsa type -1
20debug1: key_load_public: No such file or directory
21debug1: identity file /home/federico/.ssh/id_ecdsa-cert type -1
22debug1: key_load_public: No such file or directory
23debug1: identity file /home/federico/.ssh/id_ed25519 type -1
24debug1: key_load_public: No such file or directory
25debug1: identity file /home/federico/.ssh/id_ed25519-cert type -1
26debug1: Enabling compatibility mode for protocol 2.0
27debug1: Local version string SSH-2.0-OpenSSH_7.4
28debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
29debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000
30debug1: Authenticating to dumps-1.eqiad.wmflabs:22 as 'nemobis'
31debug1: SSH2_MSG_KEXINIT sent
32debug1: SSH2_MSG_KEXINIT received
33debug1: kex: algorithm: curve25519-sha256@libssh.org
34debug1: kex: host key algorithm: ecdsa-sha2-nistp256
35debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
36debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
37debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
38debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
39debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
40debug1: Server host key: ecdsa-sha2-nistp256 SHA256:mOzqp0WHr2kfsNhV+Ok3N6bj5DjQkuIViXwxwuejYsk
41debug1: Host 'dumps-1.eqiad.wmflabs' is known and matches the ECDSA host key.
42debug1: Found key in /home/federico/.ssh/known_hosts:44
43debug1: rekey after 134217728 blocks
44debug1: SSH2_MSG_NEWKEYS sent
45debug1: expecting SSH2_MSG_NEWKEYS
46debug1: SSH2_MSG_NEWKEYS received
47debug1: rekey after 134217728 blocks
48debug1: Skipping ssh-dss key /home/federico/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes
49debug1: SSH2_MSG_SERVICE_ACCEPT received
50debug1: Authentications that can continue: publickey
51debug1: Next authentication method: publickey
52debug1: Offering RSA public key: /home/federico/.ssh/id_rsa
53debug1: Server accepts key: pkalg ssh-rsa blen 279
54debug1: Authentication succeeded (publickey).
55Authenticated to dumps-1.eqiad.wmflabs (via proxy).
56debug1: channel 0: new [client-session]
57debug1: Requesting no-more-sessions@openssh.com
58debug1: Entering interactive session.
59debug1: pledge: proc
60debug1: Sending environment.
61debug1: Sending env LANG = it_IT.UTF-8
62debug1: Sending env LANGUAGE =
63Linux dumps-1 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64
64Ubuntu 14.04.5 LTS
65The last Puppet run was at Wed Jan 18 20:04:42 UTC 2017 (28 minutes ago).
66Last login: Sun Dec 18 08:54:33 2016 from bastion-01.bastion.eqiad.wmflabs
67nemobis@dumps-1:~$

Related Objects

Event Timeline

Nemo_bis created this task.Jan 29 2017, 11:16 AM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJan 29 2017, 11:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scfc subscribed.Jan 29 2017, 9:22 PM

You could look at the console log in Horizon (tab "Log") to see what is happening on the instance; if the instance's name has been used before, T147878 (and IIRC a similar bug for the migrated LDAP) comes to mind, or T148929 if it uses a custom puppetmaster. But the console log should provide something for debugging.

• bd808 subscribed.Jan 30 2017, 1:48 AM

I'm getting prompted for a password when trying to ssh to the instance using my root key as well. I would guess that the initial puppet run failed badly. Typically there is no fix for this other than deleting the instance and making another one.

Nemo_bis added a comment.Jan 30 2017, 4:37 PM

I already deleted and recreated the instance, but the result was the same.

I have no access to Horizon, so I can't check logs there.

• bd808 added a comment.Jan 30 2017, 5:15 PM
In T156580#2982337, @Nemo_bis wrote:

I have no access to Horizon, so I can't check logs there.

This is probably a tangent, but is your blocker for using Horizon two-factor auth? Other than that if you have rights to create instances via wikitech you should have access to do so via Horizon.

There should be access to instance logs for you in wikitech as well using the 'get console output' links on https://wikitech.wikimedia.org/wiki/Special:NovaInstance.

Nemo_bis added a comment.Feb 18 2017, 2:06 PM
In T156580#2982453, @bd808 wrote:
In T156580#2982337, @Nemo_bis wrote:

I have no access to Horizon, so I can't check logs there.

This is probably a tangent, but is your blocker for using Horizon two-factor auth?

Yes, the login claims it's required. Did I miss some announcement about two-factor auth being mandatory to use Labs? Let's continue at https://wikitech.wikimedia.org/w/index.php?title=Help_talk:Two-factor_authentication

There should be access to instance logs for you in wikitech as well using the 'get console output' links

Yes, I had these but the console output didn't contain anything useful. In the meanwhile all such links have disappeared, including those to create or delete instances, so I can't do any testing about this report. If instance creation was intentionally disabled on wikitech due to being broken, please close this invalid (but also write labs-announce about it).

• bd808 added a comment.Feb 18 2017, 5:02 PM
In T156580#3038135, @Nemo_bis wrote:

In the meanwhile all such links have disappeared, including those to create or delete instances, so I can't do any testing about this report. If instance creation was intentionally disabled on wikitech due to being broken, please close this invalid (but also write labs-announce about it).

https://wikitech.wikimedia.org/wiki/Special:NovaInstance still works for me. I have seen wikitech behave like the user has no rights to manage instances when the MediaWiki session and the Nova session used to communicate with OpenStack get out of sync. Logging out and back in on wikitech usually fixes this.

Nemo_bis closed this task as Invalid.EditedFeb 19 2017, 12:54 AM
Nemo_bis triaged this task as Medium priority.

True, I do see the links again now. SNAFU then, I'll keep using the logout/login trick as I've got used to across the years. I'm happy I can continue teaching this one to users. ;-)

I also manage to login in a new instance named nemobis4 with my key (though it didn't work 18 days ago when I created it, IIRC). So this report about nemobis/nemobis2 can be considered superseded.

Nemo_bis changed the task status from Invalid to Resolved.Feb 19 2017, 12:59 AM

In fact an instance I just created from Special:NovaInstance with the same name also works.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits