Page MenuHomePhabricator

beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key
Closed, ResolvedPublic


Since roughly 7:52am UTC, the beta-scap-eqiad job fails.

+ /usr/local/bin/wmf-beta-scap 'beta-scap-eqiad (build #38290)'
Starting ssh-agent
Agent pid 10854
Build step 'Execute shell' marked build as failure

Whereas a passing build has:

+ /usr/local/bin/wmf-beta-scap 'beta-scap-eqiad (build #38286)'
Starting ssh-agent
Agent pid 16027
Identity added: /var/lib/mwdeploy/.ssh/id_rsa (/var/lib/mwdeploy/.ssh/id_rsa)
Started scap: beta-scap-eqiad (build #38286)

The wmf-beta-scap wrapper runs: /usr/local/bin/sudo-withagent mwdeploy /usr/local/bin/scap

I have set +x it and it fails when invoking ssh-add which lookup for a .ssh/id_rsa in the homedir.

Seems some change has been pushed to puppet which changed the $HOME from /var/lib/mwdeploy to /home/mwdeploy hence the failure.

Event Timeline

hashar created this task.Jan 15 2015, 11:07 AM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: hashar, yuvipanda, mmodell.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2015, 11:07 AM
greg triaged this task as Unbreak Now! priority.Jan 15 2015, 4:47 PM
greg added a project: Deployments.
greg added a subscriber: greg.

Adding Deployments

@mmodell: can you take a look at this, please? @yuvipanda can probably help figure out what change broke it from ops side (if that's the case)

greg added a comment.Jan 15 2015, 4:54 PM

btw, I'm turning off the beta code update job because:
A) it's been spamming forever STILL FAILING
B) the wmf labs filesystem outage starts in 1 hour and it'd fail anyways during that

bd808 added a subscriber: bd808.Jan 15 2015, 5:02 PM
[09:57]  <    bd808>	$HOME for mwdeploy changed to /var/lib/mwdeploy quite a while ago I thought.
[09:58]  <    bd808>	oh the bug is that it changed back?
[09:58]  <    bd808>	I wonder if that has something to do with the ldap changes that Yuvi made?
[10:00]  <    bd808>	the quick hack for that is just to copy the ssh key on deployment-bastion
gerritbot added a subscriber: gerritbot.

Change 185217 had a related patch set uploaded (by Yuvipanda):
beta: Fix mwdeploy's ssh key path to point to correct path


Change 185217 merged by Yuvipanda:
beta: Fix mwdeploy's ssh key path to point to correct path

greg moved this task from To Triage to Externally Blocked on the Deployments board.
yuvipanda closed this task as Resolved.Jan 16 2015, 12:23 PM
yuvipanda claimed this task.

Unified into /home/mwdeploy now. Puppet is putting ssh keys there. It is shared across instances, but that's ok in this particular case.

I didn't notice this broke scap because I had wmf-insecte on ignore (fixed since), and I didn't notice that there was a .ssh in the homedir because I didn't check all hosts (only a couple) and didn't use -a to ls. Mea culpa.

greg moved this task from INBOX to Done on the Release-Engineering-Team board.Jan 28 2015, 10:15 PM
greg moved this task from Externally Blocked to Done on the Deployments board.Mar 11 2015, 8:48 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJun 7 2017, 6:55 PM