Page MenuHomePhabricator

strace sshd to figure out which filesystems are hit on login
Closed, ResolvedPublic

Description

ssh login was blocked, even on hosts that did not have /home on NFS. This was mostly solved by reboots. It would be good to figure out why this happened.

Event Timeline

valhallasw raised the priority of this task from to Needs Triage.
valhallasw updated the task description. (Show Details)
valhallasw added a project: Toolforge.
valhallasw subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
valhallasw removed a project: Toolforge.
valhallasw set Security to None.
chasemp claimed this task.
chasemp subscribed.

I am in agreement but since this was so long ago and there is no historical record of the incident we will have to wait for similar happenings to dig in here.

Change 256053 had a related patch set uploaded (by Merlijn van Deen):
toollabs: remove motd-tips

https://gerrit.wikimedia.org/r/256053

Change 256053 merged by Yuvipanda:
toollabs: remove motd-tips

https://gerrit.wikimedia.org/r/256053

I think this was just for tool labs, and I finally straced:

  • ps aux | grep sshd
  • strace -p <pid> -f -e trace=file,chdir 2>&1 | tee stracelog
  • grep stracelog -e '"' | sed -e 's/[^"]*";s/".*' | uniq | sort > files

which showed we access /data/project/.system/tips, which was probably the underlying cause of this issue.

valhallasw changed the task status from Invalid to Resolved.Nov 30 2015, 7:30 PM

Change 260734 had a related patch set uploaded (by Yuvipanda):
ldap: Set home for the LDAP lookup user

https://gerrit.wikimedia.org/r/260734

Change 260734 merged by Faidon Liambotis:
ldap: Set home for the LDAP lookup user

https://gerrit.wikimedia.org/r/260734