Running apt-get update on any of the toolsbeta buster sgeexec nodes fails as apt fails to write to /tmp:
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease: Couldn't create temporary file /tmp/apt.conf.LCSA4F for passing config to apt-key
/tmp is mounted from a separate drive:
root@toolsbeta-sgeexec-1002:/tmp# df -h|grep /tmp /dev/sdc 20G 205M 20G 1% /tmp
Running ls on the directory stalls, so it seems like something is wrong with the mount itself?
Trying to manually unmount and re-mount the drive fails:
root@toolsbeta-sgeexec-1002:~# umount /tmp umount: /tmp: target is busy.