make deployment SSH keys use the same passphrase
Closed, ResolvedPublic

Description

Make all SSH keys used for deployment and loaded by keyholder use the same passphrase instead of different ones.

Update pwstore and https://wikitech.wikimedia.org/wiki/Keyholder

Dzahn created this task.Jan 9 2017, 9:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2017, 9:12 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-20T00:46:55Z] <mutante> setting all deployment key passphrases to the one used for mw deploy - update key files in private repo (T154943)

Mentioned in SAL (#wikimedia-operations) [2017-01-20T00:49:58Z] <mutante> mira - arming keyholder after setting service/dumps/eventlogging/phabricator key passphrases to the same one (T154943)

Dzahn added a comment.Jan 20 2017, 6:32 AM

Reverted to the old keys for the moment. When changing passphrase with ssh-keygen -p -f .. a side-effect was that ssh-add -l does not show full path anymore as default comment, only shows "rsa w/o comment". But keyholder/check_keyholder relies on those pathes in the ssh-add output. I talked in #openssh and with volans. Will try again and add detailed commands.

Dzahn added a comment.Jan 20 2017, 6:46 AM
17:54 < mutante> !log tin - keyholder disarm and arm again using new passphrase
16:51 < mutante> !log mira - arming keyholder after setting service/dumps/eventlogging/phabricator key passphrases to the same one 
                 (T154943)

16:56 < icinga-wm> PROBLEM - Keyholder SSH agent on tin is CRITICAL: CRITICAL: Keyholder is not armed. Run keyholder arm to arm it.


16:58 < mutante> oh come on, it's totally armed per "status"
17:01 < volans> mutante: the paths are missing
17:02 < volans> no sorry
17:02 < mutante> volans: you mean in the output of "keyholder status"? that's just the key comments
17:02 < mutante> which are not set
17:02 < volans> mutante: I thought was that but actually is the ssh-add -l that is empty

17:03 < volans> the command in /usr/lib/nagios/plugins/check_keyholder applied to the outout of ssh-add is returning just rsa
17:03 < volans> because that should be the path
17:04 < volans> and it doesnt' match with the sorted paths in configured_keys() that are taken from a find

17:08 < mutante> volans: hmm, i dont get why analytics would be the only one different
17:08 < mutante> thanks, so i guess i removed the key comment by just not providing it
17:09 < volans> mutante: yeah, seems so
17:09 < mutante> but weird how one key kept it

17:26 < mutante> volans: "Comments are only supported for RSA1 keys.
17:27 < mutante> so even with -C / -c not adding the comment silently
17:29 < volans> mutante: how did they have the comment before?
17:29 < volans> maybe rsa1 is the only one that allow to change it?
17:32 < mutante>  This operation is only supported for RSA1 keys. 

17:32 < volans> adding them manually in the public key is not enough I guess... :-P

...

18:31 < icinga-wm> RECOVERY - Keyholder SSH agent on mira is OK: OK: Keyholder is armed with all configured keys.
18:32 < volans> mutante: \o/
18:33 < mutante> yea, but it's only a revert :/
18:33 < mutante> i talked in #openssh for a while even
18:33 < mutante> they say old ssh-add version .. but .. eh
18:34 < mutante> and that's not the comment line, that's just the full path to the key
18:34 < mutante> https://paste.pound-python.org/show/UOsCwU9cRuEPVdLKcLxm/  etc
18:34 < volans> that disappear when changing the password?
18:34 < mutante> yes
18:34 < mutante> except for one of them
18:34 < volans> WAT?!?!

18:35 < mutante> 18:18 < BasketCase> I am not sure why manual ssh-add doesn't add a comment
18:35 < mutante> 18:19 < BasketCase> also, manual ssh-add actually added the comment from the pub file for my ed25519 key but not my rsa key
18:35 < mutante> 18:22 < BasketCase> I am on 7.3 btw
18:35 < mutante> 18:22 < BasketCase> not whatever obsolete junk Debian has

18:36 < mutante> yea, eh, i am reverting to get back to it later
18:36 < mutante> totally unexpected rabbit hole here
18:36 < volans> ok, otherwise we could just create new ones
18:36 < mutante> yes, true
18:37 < volans> probably easier and also safer, as an not-scheduled key rotation :)

18:39 < icinga-wm> RECOVERY - Keyholder SSH agent on tin is OK: OK: Keyholder is armed with all configured keys.

18:14 < mutante> man ssh-keygen -c Requests changing the comment in the private and public key files. "
18:14 < BasketCase> at least it is on my system
18:14 < BasketCase> yes, v1 keys had an embedded comment
18:14 < mutante> 4096 35:c7:36:72:0f:a9:9e:0c:a8:d7:bb:5e:bf:17:9b:94 /etc/keyholder.d/analytics_deploy (RSA)
18:14 < mutante> 2048 6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 rsa w/o comment (RSA)
18:15 < mutante> i have this stuff
18:15 < mutante> see the "w/o comment" part
18:15 < mutante> that is my problem
18:15 < BasketCase> your version is old
18:16 < mutante> so i have to create new keys?
18:16 < mutante> still doesnt get how he removed the existing comments 
18:16 < BasketCase> no, your ssh-add command is old
18:16 < mutante> while changing passphrase
18:16 < BasketCase> it is still using an md5 hash
18:16 < BasketCase> when I look deeper, if I manually add a key to my agent it says there is no comment like that
18:17 < BasketCase> my keys added via pam_ssh have the file name as the comment
18:17 < BasketCase> neither is the comment stored in the file

18:17 < mutante> hmm.. i need the file name in the comment for all of them, yes

18:18 < BasketCase> I am not sure why manual ssh-add doesn't add a comment
18:19 < BasketCase> also, manual ssh-add actually added the comment from the pub file for my ed25519 key but not my rsa key

18:20 < mutante> ugh, did _not_ expect these changes
18:22 < BasketCase> https://paste.pound-python.org/show/UOsCwU9cRuEPVdLKcLxm/
18:22 < BasketCase> I am on 7.3 btw
18:22 < BasketCase> not whatever obsolete junk Debian has
18:24 < BasketCase> I have no idea why ed25519 was treated differently there
18:24 < mutante> i see. thanks.. hmm
18:25 < BasketCase> but the agent I get from pam_ssh shows this: https://paste.pound-python.org/show/p7xcGzVOzLOzKlWoEVEG/
18:25 < BasketCase> none of those are the real comment but they are something specific at least
18:26  * BasketCase loves pam_ssh

18:28 < BasketCase> if your ssh key fingerprints are still md5 I am not sure if my suggestion of ssh-keygen -o will work
18:29 < BasketCase> I think the new format came before a secure hash but I don't feel like looking it up to make sure
18:29 < mutante> ok, for now i am taking the files before i touched them at all
18:29 < BasketCase> if you have ed25519 support you have ssh-keygen -o
18:31 < mutante> so after reverting to the unchanged keyfiles i have the file pathes back 
18:31 < mutante> for all of them. same ssh-keygen
18:33 < BasketCase> did you do the new format transition I was suggesting when you upgraded?  maybe the current behavior doesn't apply to the old format?
18:34 < BasketCase> the file command would identify the private key files as different types
18:36 < BasketCase> "PEM RSA private key" is the old format "OpenSSH private key" is the new format
18:36 < BasketCase> (assuming a modern file magic)
18:39 < mutante> i did not to that yet, no
18:39 < mutante> i think i'll create new keys tomorrow

18:39 < BasketCase> then I am not sure why behavior changed
18:40 < BasketCase> the new format is much more resistant to brute force attacks btw

18:40 < BasketCase> essentially, the old kdf was just md5
18:41 < BasketCase> so the encryption key was just an md5 hash

Change 312947 had a related patch set uploaded (by Dzahn):
Fix failing keyholder arming check

https://gerrit.wikimedia.org/r/312947

Change 312947 merged by Dzahn:
Fix failing keyholder arming check

https://gerrit.wikimedia.org/r/312947

Mentioned in SAL (#wikimedia-operations) [2017-03-02T00:23:10Z] <mutante> mira - disarming keyholder, changed password of analytics deploy key - rearming to test changes for T154943

Mentioned in SAL (#wikimedia-operations) [2017-03-02T00:41:14Z] <mutante> mira - disarm/rearm keyholder after changing passphrases of all other deployment keys (T154943)

Mentioned in SAL (#wikimedia-operations) [2017-03-02T00:44:41Z] <mutante> tin - disarm/rearm keyholder after changing passphrases of all deployment keys to new passphrase (T154943)

Dzahn added a comment.Thu, Mar 2, 1:13 AM
  • dis-armed and re-armed keyholder on both mira and tin, confirming that you are asked just once for the new passphrase and all keys are loaded
  • you will notice in the output of keyholder status you do not see the file names in the "comment" column anymore [1]. This is due to newer versions of openssh-client and caused all the debugging last time (T154943#2955275). meanwhile, thanks to thcipriani's fix (https://gerrit.wikimedia.org/r/#/c/312947/) we don't rely on the file names but on the fingerprints and these are unchanged. keys just got re-encrypted with new passphrase with ssh-keygen -p.
  • added file deployment-key-passphrase to pwstore containing the new unified passphrase
  • updated wikitech page accordingly: https://wikitech.wikimedia.org/w/index.php?title=Keyholder&type=revision&diff=1618083&oldid=1548651

[1]

before:

[tin:~] $ sudo keyholder status
keyholder-agent: active
..
- 2048 6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 /etc/keyholder.d/deploy_service (RSA)
- 2048 86:c9:17:ab:b7:00:79:b5:8a:c5:b5:ee:29:24:c9:2f /etc/keyholder.d/dumpsdeploy (RSA)

after:

[tin:~] $ sudo keyholder status
keyholder-agent: active
..
- 2048 6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 rsa w/o comment (RSA)
- 2048 86:c9:17:ab:b7:00:79:b5:8a:c5:b5:ee:29:24:c9:2f rsa w/o comment (RSA)
..
Dzahn added a subscriber: Volans.Thu, Mar 2, 1:30 AM
  • reverted the cumin key back to like it was before as requested by @Volans who pointed out this should keep having a separate passphrase
Dzahn closed this task as "Resolved".Thu, Mar 2, 10:27 PM

sent mail about this to ops list. i consider this resolved now.