Some random puppet change has been pushed early today around 8:00am UTC which caused the Beta cluster scap to fails.
The root cause is that mwdeploy homedir is /home/mwdeploy but should be /var/lib/mwdeploy , thus the scap script wrapper fails to ssh-add because it can't find the ssh key.
$ getent passwd mwdeploy mwdeploy:x:603:603:mwdeploy:/home/mwdeploy:/bin/bash $ ldaplist -l passwd mwdeploy|grep homeDirectory homeDirectory: /home/mwdeploy $
We would need to update the mwdeploy homeDirectory in LDAP to point to /var/lib/mwdeploy. We did the same for the jenkins-deploy user used by Jenkins slaves:
$ ldaplist -l passwd jenkins-deploy|grep homeDirectory homeDirectory: /mnt/home/jenkins-deploy