Page MenuHomePhabricator

Production access has been approved but not able to log in, access was a long time ago so it's a new problem
Closed, ResolvedPublic

Assigned To
Authored By
MBinder_WMF
Sep 11 2024, 10:25 PM
Referenced Files
F57503894: image.png
Sep 12 2024, 9:56 PM
F57503878: config
Sep 12 2024, 9:41 PM
F57500778: config
Sep 11 2024, 10:56 PM

Description

Hi!

I'm following a protocol here: https://wikitech.wikimedia.org/wiki/SRE/Production_access#Debugging

I was originally granted access here: https://phabricator.wikimedia.org/T251349

It's been a long time since I tried to log in, so I filed a new ticket. :)

At first, I tried to SSH into phab1001.eqiad.wmnet, and got

ssh: Could not resolve hostname bast4003.wikimedia.org: nodename nor servname provided, or not known
Connection closed by UNKNOWN port 65535

I think I resolved that by updating my config file to a bastion listed here: https://wikitech.wikimedia.org/wiki/Bastion

I was prompted for a password, indicating an issue with my config file.

I tried phab1001.eqiad.wmnet and bast1002.eqiad.wmnet.

Here's my verbose output for the latter (in a restricted doc just because I don't fully understand the output): https://docs.google.com/document/d/1W3JV0zKcAOwgo63NLIyDUM-en2LP8XcLtjxd_kB_tng/edit

My config file is identical to the one at https://wikitech.wikimedia.org/wiki/SRE/Production_access#Setting_up_your_SSH_config except for

  • username
  • the name of my identity file

I can share the config file once someone confirms that's safe to do (I'm just being extra careful by default).

Event Timeline

config file attached after confirming with @Ladsgroup

I'm guessing but:

  • Instead of bast1002 or bast4003, use bast4005.wikimedia.org (depending on where you live). Otherwise, it'll be quite slow.
  • phab1001 I think has been decommissioned long time ago. the Phabricator hosts that is serving us is phab1004. Can you try that?

I tried phab1001.eqiad.wmnet and bast1002.eqiad.wmnet.

Hi! The issue here is that these host names are outdated.

Phabricator (Phorge) currently runs on phab1004.eqiad.wmnet.

https://wikitech.wikimedia.org/wiki/Phabricator

The current bastion host in eqiad is bast1003.eqiad.wmnet (or you can use any other bastion from the page below).

https://wikitech.wikimedia.org/wiki/Bastion

Host names change every time we replace hardware and it's been a long time as you say.

Your access should still work, just on the new host names. Let us know if that fixes it.

Thanks, both. I tried to ssh into phab1004.eqiad.wmnet with bast1003.wikimedia.org in the config file, and got the same issue.

I tried to ssh into phab1004.eqiad.wmnet with bast1003.eqiad.wmnet in the config file, and had a long delay before the passphrase prompt occurred again.

I tried to ssh directly to bast1003.eqiad.wmnet with bast1003.eqiad.wmnet in the config file, and had the same delay.

I've added a verbose output to the doc. The verbose output for the option that was delayed is too long for copy/paste to handle (it seems like a loop). I was able to export a txt file, though, and can upload that if you need. For some reason, the first time I got the delay, I got to the passphrase stage. The second time, I got disconnected. I'm not sure if I did something differently.

The correct combination is phab1004.eqiad.wmnet with bast1003.wikimedia.org.

The "bast" hosts have the public IPs and end in wikimedia.org. Other hosts with private IP are "behind" them and end in .wmnet.

Start testing by trying ssh direct to bast1003.wikimedia.org and make sure the user is mbinder.

So ssh mbinder@bast1003.wikimedia.org should get you a login on the bast host itself.

If that works with the key you are using / have configured then we can get to the second step, which is jumping via this host to the phab host behind it.

Right now I can see that you connected to bast1003 but never tried to login on phab1004.

So it seems like the issue is now in the local ssh config.

Could you paste the contents of /Users/maxbinder/.ssh/config ?

The correct combination is phab1004.eqiad.wmnet with bast1003.wikimedia.org.

Attached is my verbose output for that combination.

So ssh mbinder@bast1003.wikimedia.org should get you a login on the bast host itself.

Still prompted for a passphrase. Here's the verbose output:

Config file:

Ah, I think I might know the problem: my public key file specifies the name of a computer that preceded my current one. I was pretty sure I had logged in on the new machine, but I can't find a record of that, so it's possible I need to generate a new key pair.

Do you have a file /Users/maxbinder/.ssh/id_ed25519.pub ? (not /Users/maxbinder/.ssh/id_ed25519 the private part, just the public part ending in .pub)?

And if so, does it look like this?

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFLrpbK99cmNng3y8t1qYOvTKksI4TZKtQSTsF9RZ1pC maxbinder@Maxs-MacBook-Air.local[

The comment at the end should not matter, but the rest should match.

Also, try this:

Move the config file out of the way temporarily.

Like mv /Users/maxbinder/.ssh/config /Users/maxbinder/ so it does not get used. (later move it back).

Then do a direct ssh command where we tell it specifically which key to use and which user name to use:

ssh -i /Users/maxbinder/.ssh/id_ed25519 mbinder@bast1003.wikimedia.org

Let's see the output of that and if you get a shell with that.

Do you have a file /Users/maxbinder/.ssh/id_ed25519.pub ? (not /Users/maxbinder/.ssh/id_ed25519 the private part, just the public part ending in .pub)?

And if so, does it look like this?

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFLrpbK99cmNng3y8t1qYOvTKksI4TZKtQSTsF9RZ1pC maxbinder@Maxs-MacBook-Air.local[

The comment at the end should not matter, but the rest should match.

Yes, here's a screenshot of the location and contents:

image.png (528×2 px, 289 KB)

You shouldn't have to create a keypair just because your computer name changed. The part at the end is mostly just a comment field.

Also, try this:

Move the config file out of the way temporarily.

Like mv /Users/maxbinder/.ssh/config /Users/maxbinder/ so it does not get used. (later move it back).

Then do a direct ssh command where we tell it specifically which key to use and which user name to use:

ssh -i /Users/maxbinder/.ssh/id_ed25519 mbinder@bast1003.wikimedia.org

Let's see the output of that and if you get a shell with that.

Output still asked for passphrase:

Can you run the ssh command with -vvvvvvvv (the more "v"s, the better)?

but share the result privately, just in case.

Ok, do this:

ssh-add /Users/maxbinder/.ssh/id_ed25519

It should just ask for a passphrase. If you know it, enter it.

Now that key should be loaded in the background and you are back at the shell. You can verify the key is loaded with ssh-add -l.

Now try ssh mbinder@bast1003.wikimedia.org again.

but share the result privately, just in case.

doc updated with many v's :)

Ok, do this:

ssh-add /Users/maxbinder/.ssh/id_ed25519

It should just ask for a passphrase. If you know it, enter it.

Now that key should be loaded in the background and you are back at the shell. You can verify the key is loaded with ssh-add -l.

Now try ssh mbinder@bast1003.wikimedia.org again.

worked!

I can successfully log on to phab1004.eqiad.wmnet as well. What was the issue?

Output still asked for passphrase:

So that is the thing, the passphrase is needed to decrypt the key itself.

Now that it's loaded in the background it won't ask you again and again for it. Until you reboot next time, then you have to add it back to the agent (with that ssh-add command).

FWIW, ssh -vvvvvvvvvvvvvvvvvvvv ~/.ssh/id_ed25519 mbinder@bast1003.wikimedia.org broke because:

ssh: Could not resolve hostname /users/maxbinder/.ssh/id_ed25519: nodename nor servname provided, or not known

It was trying to connect to /users/maxbinder/.ssh/id_ed25519 instead of mbinder@bast1003.wikimedia.org (wrong order of arguments or it shouldn't even take path to private key as a positional argument?)

I can successfully log on to phab1004.eqiad.wmnet as well. What was the issue?

The key is used to login on the server. Not a password.

But also the key is encrypted so that it's not so easy to steal it from you.

So to use the key you have to first decrypt it. That is what that passphrase is.

There are basically 2 options. Either you enter it every single time you connect to a server, or you load it into the so called "agent" so that it stays loaded in the background (for this session on your local computer), which makes it a little more convenient.

The ssh-add command is "decrypt that key and load it into the agent".

Hmm, I'm pretty sure I never had to enter a passphrase for each login in the past, but I might be mistaken. Also, when I was prompted the first time for a passphrase, I tried the same one on a whim, and it didn't work. That doesn't mean this is wrong! But it's curious that it wasn't accepted at that stage (if it was I never would have asked for support).

This probably has to do with getting your new computer. Likely you had this key added to some kind of key chain or app provided by the OS that always loaded it for you automatically.

This probably has to do with getting your new computer. Likely you had this key added to some kind of key chain or app provided by the OS that always loaded it for you automatically.

But wouldn't the prompt for the password want the same input? Presumably, if I restart my machine now, and try to ssh without decrypting first, I will be prompted for a passphrase. If I put in the same one I did for decryption, it should work no?

Ah, you know what? I think it did, in fact, work. I just didn't realize that I needed to enter it twice, and assumed that the re-prompt was a failure and closed the terminal without trying again. Then, it occurred to me I might have typed it wrong (since it's invisible), re-opened the terminal to try again, and even though I was careful to type it right I still thought the re-prompt meant failure. Probably could have saved a lot of trouble just trying it twice in the same window!

Anyway, it works now both ways. Thanks for your assistance and patience. :)

Ladsgroup assigned this task to MBinder_WMF.

Don't know Mac but in Linux you can set it to "Remember the key passphrase" and it would remember it for a long time. Regardless, glad it's resolved \o/