beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Jan 15 2015, 11:07 AM

Description

Since roughly 7:52am UTC, the beta-scap-eqiad job fails.

+ /usr/local/bin/wmf-beta-scap 'beta-scap-eqiad (build #38290)'
Starting ssh-agent
Agent pid 10854
Build step 'Execute shell' marked build as failure

Whereas a passing build has:

+ /usr/local/bin/wmf-beta-scap 'beta-scap-eqiad (build #38286)'
Starting ssh-agent
Agent pid 16027
Identity added: /var/lib/mwdeploy/.ssh/id_rsa (/var/lib/mwdeploy/.ssh/id_rsa)
Started scap: beta-scap-eqiad (build #38286)[0m

The wmf-beta-scap wrapper runs: /usr/local/bin/sudo-withagent mwdeploy /usr/local/bin/scap

I have set +x it and it fails when invoking ssh-add which lookup for a .ssh/id_rsa in the homedir.

Seems some change has been pushed to puppet which changed the $HOME from /var/lib/mwdeploy to /home/mwdeploy hence the failure.

Details

	Subject	Repo	Branch	Lines +/-
	beta: Fix mwdeploy's ssh key path to point to correct path	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		yuvipanda	T86901 beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key
		Declined		yuvipanda	T86903 Change mwdeploy homeDirectory field in LDAP from /home/mwdeploy to /var/lib/mwdeploy

Event Timeline

hashar created this task.Jan 15 2015, 11:07 AM

hashar raised the priority of this task from to Needs Triage.

hashar updated the task description. (Show Details)

hashar added projects: Beta-Cluster-Infrastructure, Release-Engineering-Team.

hashar added subscribers: hashar, yuvipanda, • mmodell.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2015, 11:07 AM

hashar mentioned this in T86668: Make all ldap users have a sane shell (/bin/bash).Jan 15 2015, 11:16 AM

Adding Deployments

@mmodell: can you take a look at this, please? @yuvipanda can probably help figure out what change broke it from ops side (if that's the case)

btw, I'm turning off the beta code update job because:
A) it's been spamming forever STILL FAILING
B) the wmf labs filesystem outage starts in 1 hour and it'd fail anyways during that

[09:57]  <    bd808>	$HOME for mwdeploy changed to /var/lib/mwdeploy quite a while ago I thought.
[09:58]  <    bd808>	oh the bug is that it changed back?
[09:58]  <    bd808>	I wonder if that has something to do with the ldap changes that Yuvi made?
[10:00]  <    bd808>	the quick hack for that is just to copy the ssh key on deployment-bastion

Change 185217 had a related patch set uploaded (by Yuvipanda):
beta: Fix mwdeploy's ssh key path to point to correct path

https://gerrit.wikimedia.org/r/185217

Patch-For-Review

yuvipanda closed subtask T86903: Change mwdeploy homeDirectory field in LDAP from /home/mwdeploy to /var/lib/mwdeploy as Declined.Jan 15 2015, 6:39 PM

Change 185217 merged by Yuvipanda:
beta: Fix mwdeploy's ssh key path to point to correct path

https://gerrit.wikimedia.org/r/185217

yuvipanda mentioned this in rOPUPd6fe35058e71: beta: Fix mwdeploy's ssh key path to point to correct path.Jan 15 2015, 7:32 PM

greg moved this task from To Triage to In-progress on the Beta-Cluster-Infrastructure board.Jan 15 2015, 9:28 PM

greg moved this task from To Triage to Externally Blocked on the Deployments board.

Unified into /home/mwdeploy now. Puppet is putting ssh keys there. It is shared across instances, but that's ok in this particular case.

I didn't notice this broke scap because I had wmf-insecte on ignore (fixed since), and I didn't notice that there was a .ssh in the homedir because I didn't check all hosts (only a couple) and didn't use -a to ls. Mea culpa.

greg moved this task from In-progress to Done on the Beta-Cluster-Infrastructure board.Jan 17 2015, 6:34 PM

greg moved this task from INBOX to Done on the Release-Engineering-Team board.Jan 28 2015, 10:15 PM

Jdforrester-WMF removed a project: Patch-For-Review.Mar 3 2015, 5:27 PM

greg moved this task from Externally Blocked to Done on the Deployments board.Mar 11 2015, 8:48 PM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:55 PM

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJun 7 2017, 6:55 PM

beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh keyClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

beta-scap-eqiad fails due to ssh-add not finding mwdeploy ssh key
Closed, ResolvedPublic
Actions

Related Objects
Search...