Reverted to the old keys for the moment. When changing passphrase with ssh-keygen -p -f .. a side-effect was that ssh-add -l does not show full path anymore as default comment, only shows "rsa w/o comment". But keyholder/check_keyholder relies on those pathes in the ssh-add output. I talked in #openssh and with volans. Will try again and add detailed commands.
17:54 < mutante> !log tin - keyholder disarm and arm again using new passphrase 16:51 < mutante> !log mira - arming keyholder after setting service/dumps/eventlogging/phabricator key passphrases to the same one (T154943) 16:56 < icinga-wm> PROBLEM - Keyholder SSH agent on tin is CRITICAL: CRITICAL: Keyholder is not armed. Run keyholder arm to arm it. 16:58 < mutante> oh come on, it's totally armed per "status" 17:01 < volans> mutante: the paths are missing 17:02 < volans> no sorry 17:02 < mutante> volans: you mean in the output of "keyholder status"? that's just the key comments 17:02 < mutante> which are not set 17:02 < volans> mutante: I thought was that but actually is the ssh-add -l that is empty 17:03 < volans> the command in /usr/lib/nagios/plugins/check_keyholder applied to the outout of ssh-add is returning just rsa 17:03 < volans> because that should be the path 17:04 < volans> and it doesnt' match with the sorted paths in configured_keys() that are taken from a find 17:08 < mutante> volans: hmm, i dont get why analytics would be the only one different 17:08 < mutante> thanks, so i guess i removed the key comment by just not providing it 17:09 < volans> mutante: yeah, seems so 17:09 < mutante> but weird how one key kept it 17:26 < mutante> volans: "Comments are only supported for RSA1 keys. 17:27 < mutante> so even with -C / -c not adding the comment silently 17:29 < volans> mutante: how did they have the comment before? 17:29 < volans> maybe rsa1 is the only one that allow to change it? 17:32 < mutante> This operation is only supported for RSA1 keys. 17:32 < volans> adding them manually in the public key is not enough I guess... :-P ... 18:31 < icinga-wm> RECOVERY - Keyholder SSH agent on mira is OK: OK: Keyholder is armed with all configured keys. 18:32 < volans> mutante: \o/ 18:33 < mutante> yea, but it's only a revert :/ 18:33 < mutante> i talked in #openssh for a while even 18:33 < mutante> they say old ssh-add version .. but .. eh 18:34 < mutante> and that's not the comment line, that's just the full path to the key 18:34 < mutante> https://paste.pound-python.org/show/UOsCwU9cRuEPVdLKcLxm/ etc 18:34 < volans> that disappear when changing the password? 18:34 < mutante> yes 18:34 < mutante> except for one of them 18:34 < volans> WAT?!?! 18:35 < mutante> 18:18 < BasketCase> I am not sure why manual ssh-add doesn't add a comment 18:35 < mutante> 18:19 < BasketCase> also, manual ssh-add actually added the comment from the pub file for my ed25519 key but not my rsa key 18:35 < mutante> 18:22 < BasketCase> I am on 7.3 btw 18:35 < mutante> 18:22 < BasketCase> not whatever obsolete junk Debian has 18:36 < mutante> yea, eh, i am reverting to get back to it later 18:36 < mutante> totally unexpected rabbit hole here 18:36 < volans> ok, otherwise we could just create new ones 18:36 < mutante> yes, true 18:37 < volans> probably easier and also safer, as an not-scheduled key rotation :) 18:39 < icinga-wm> RECOVERY - Keyholder SSH agent on tin is OK: OK: Keyholder is armed with all configured keys. 18:14 < mutante> man ssh-keygen -c Requests changing the comment in the private and public key files. " 18:14 < BasketCase> at least it is on my system 18:14 < BasketCase> yes, v1 keys had an embedded comment 18:14 < mutante> 4096 35:c7:36:72:0f:a9:9e:0c:a8:d7:bb:5e:bf:17:9b:94 /etc/keyholder.d/analytics_deploy (RSA) 18:14 < mutante> 2048 6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 rsa w/o comment (RSA) 18:15 < mutante> i have this stuff 18:15 < mutante> see the "w/o comment" part 18:15 < mutante> that is my problem 18:15 < BasketCase> your version is old 18:16 < mutante> so i have to create new keys? 18:16 < mutante> still doesnt get how he removed the existing comments 18:16 < BasketCase> no, your ssh-add command is old 18:16 < mutante> while changing passphrase 18:16 < BasketCase> it is still using an md5 hash 18:16 < BasketCase> when I look deeper, if I manually add a key to my agent it says there is no comment like that 18:17 < BasketCase> my keys added via pam_ssh have the file name as the comment 18:17 < BasketCase> neither is the comment stored in the file 18:17 < mutante> hmm.. i need the file name in the comment for all of them, yes 18:18 < BasketCase> I am not sure why manual ssh-add doesn't add a comment 18:19 < BasketCase> also, manual ssh-add actually added the comment from the pub file for my ed25519 key but not my rsa key 18:20 < mutante> ugh, did _not_ expect these changes 18:22 < BasketCase> https://paste.pound-python.org/show/UOsCwU9cRuEPVdLKcLxm/ 18:22 < BasketCase> I am on 7.3 btw 18:22 < BasketCase> not whatever obsolete junk Debian has 18:24 < BasketCase> I have no idea why ed25519 was treated differently there 18:24 < mutante> i see. thanks.. hmm 18:25 < BasketCase> but the agent I get from pam_ssh shows this: https://paste.pound-python.org/show/p7xcGzVOzLOzKlWoEVEG/ 18:25 < BasketCase> none of those are the real comment but they are something specific at least 18:26 * BasketCase loves pam_ssh 18:28 < BasketCase> if your ssh key fingerprints are still md5 I am not sure if my suggestion of ssh-keygen -o will work 18:29 < BasketCase> I think the new format came before a secure hash but I don't feel like looking it up to make sure 18:29 < mutante> ok, for now i am taking the files before i touched them at all 18:29 < BasketCase> if you have ed25519 support you have ssh-keygen -o 18:31 < mutante> so after reverting to the unchanged keyfiles i have the file pathes back 18:31 < mutante> for all of them. same ssh-keygen 18:33 < BasketCase> did you do the new format transition I was suggesting when you upgraded? maybe the current behavior doesn't apply to the old format? 18:34 < BasketCase> the file command would identify the private key files as different types 18:36 < BasketCase> "PEM RSA private key" is the old format "OpenSSH private key" is the new format 18:36 < BasketCase> (assuming a modern file magic) 18:39 < mutante> i did not to that yet, no 18:39 < mutante> i think i'll create new keys tomorrow 18:39 < BasketCase> then I am not sure why behavior changed 18:40 < BasketCase> the new format is much more resistant to brute force attacks btw 18:40 < BasketCase> essentially, the old kdf was just md5 18:41 < BasketCase> so the encryption key was just an md5 hash
- dis-armed and re-armed keyholder on both mira and tin, confirming that you are asked just once for the new passphrase and all keys are loaded
- you will notice in the output of keyholder status you do not see the file names in the "comment" column anymore . This is due to newer versions of openssh-client and caused all the debugging last time (T154943#2955275). meanwhile, thanks to thcipriani's fix (https://gerrit.wikimedia.org/r/#/c/312947/) we don't rely on the file names but on the fingerprints and these are unchanged. keys just got re-encrypted with new passphrase with ssh-keygen -p.
- added file deployment-key-passphrase to pwstore containing the new unified passphrase
- updated wikitech page accordingly: https://wikitech.wikimedia.org/w/index.php?title=Keyholder&type=revision&diff=1618083&oldid=1548651
[tin:~] $ sudo keyholder status keyholder-agent: active .. - 2048 6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 /etc/keyholder.d/deploy_service (RSA) - 2048 86:c9:17:ab:b7:00:79:b5:8a:c5:b5:ee:29:24:c9:2f /etc/keyholder.d/dumpsdeploy (RSA)
[tin:~] $ sudo keyholder status keyholder-agent: active .. - 2048 6d:54:92:8b:39:10:f5:9b:84:40:36:ef:3c:9a:6d:d8 rsa w/o comment (RSA) - 2048 86:c9:17:ab:b7:00:79:b5:8a:c5:b5:ee:29:24:c9:2f rsa w/o comment (RSA) ..