ssh login was blocked, even on hosts that did not have /home on NFS. This was mostly solved by reboots. It would be good to figure out why this happened.
I think this was just for tool labs, and I finally straced:
- ps aux | grep sshd
- strace -p <pid> -f -e trace=file,chdir 2>&1 | tee stracelog
- grep stracelog -e '"' | sed -e 's/[^"]*";s/".*' | uniq | sort > files
which showed we access /data/project/.system/tips, which was probably the underlying cause of this issue.