While deploying dumps-nfs.w.o mounted to /mnt/nfs/dumps I ran into an issue I didn't consider initially: upon failover we're umounting /mnt/nfs/dumps whenever the nfs client starts returning ESTALE. This works fine within the host, however containers that bind-mount the directory will not see the umount by default.
Linux mount namespaces support the concept of propagating mount events for bind mounts, this feature is exposed as https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation HostToContainer by k8s and rslave mount option.
The catch is that propagation works for subdirectories under the bind-mount, not for the bind-mount itself, in other words we need to mount a subdirectory of the actual mountpoint within the container. My first instinct was to bind-mount /mnt/nfs though I believe that's a non-starter because it also contains home/project mounts? I hope I'm wrong though.
The alternative of course is to establish sth like /mnt/nfs/dumps/mount and then bind-mount /mnt/nfs/dumps.
Reproducer to illustrate the behavior of parent directory vs exact bind mount
setup
mkdir -p /root/proptest/nfs mount --bind /root/proptest /root/proptest mount --make-shared /root/proptest mount -t tmpfs nfs-sim /root/proptest/nfs echo "original content" > /root/proptest/nfs/marker.txt apt install containerd ctr image pull docker.io/library/alpine:latest
bind mount rslave the parent directory
ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propA sleep infinity ctr task exec --exec-id t1 propA cat /data/nfs/marker.txt # remount, NEW content will be shown umount /root/proptest/nfs mount -t tmpfs nfs-sim2 /root/proptest/nfs echo "NEW content after remount" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t2 propA cat /data/nfs/marker.txt
bind mount the exact mountpoint, content is not propagated
ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest/nfs,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propB sleep infinity ctr task exec --exec-id t3 propB cat /data/marker.txt # remount umount /root/proptest/nfs mount -t tmpfs nfs-sim3 /root/proptest/nfs echo "EVEN NEWER content" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t4 propB cat /data/marker.txt
deployment strategy
- get new volume-admission with mountpropagation hosttocontainer and bind-mount /mnt/nfs
- deploy to toolsbeta, enable dumps-nfs
- restart all pods to pick up the changes
- verify functionality
- repeat for tools, possibly announcing the change