Page MenuHomePhabricator

strace sshd to figure out which filesystems are hit on login
Closed, ResolvedPublic

Description

ssh login was blocked, even on hosts that did not have /home on NFS. This was mostly solved by reboots. It would be good to figure out why this happened.

Event Timeline

valhallasw raised the priority of this task from to Needs Triage.
valhallasw updated the task description. (Show Details)
valhallasw added a project: Toolforge.
valhallasw added a subscriber: valhallasw.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJun 30 2015, 12:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
valhallasw updated the task description. (Show Details)Jun 30 2015, 12:42 PM
valhallasw removed a project: Toolforge.
valhallasw set Security to None.
chasemp closed this task as Invalid.Nov 30 2015, 4:33 PM
chasemp claimed this task.
chasemp added a subscriber: chasemp.

I am in agreement but since this was so long ago and there is no historical record of the incident we will have to wait for similar happenings to dig in here.

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 30 2015, 4:33 PM

Change 256053 had a related patch set uploaded (by Merlijn van Deen):
toollabs: remove motd-tips

https://gerrit.wikimedia.org/r/256053

Change 256053 merged by Yuvipanda:
toollabs: remove motd-tips

https://gerrit.wikimedia.org/r/256053

I think this was just for tool labs, and I finally straced:

  • ps aux | grep sshd
  • strace -p <pid> -f -e trace=file,chdir 2>&1 | tee stracelog
  • grep stracelog -e '"' | sed -e 's/[^"]*";s/".*' | uniq | sort > files

which showed we access /data/project/.system/tips, which was probably the underlying cause of this issue.

valhallasw changed the task status from Invalid to Resolved.Nov 30 2015, 7:30 PM

cool great news

Change 260734 had a related patch set uploaded (by Yuvipanda):
ldap: Set home for the LDAP lookup user

https://gerrit.wikimedia.org/r/260734

Change 260734 merged by Faidon Liambotis:
ldap: Set home for the LDAP lookup user

https://gerrit.wikimedia.org/r/260734