Change Details

While deploying `dumps-nfs.w.o` mounted to `/mnt/nfs/dumps` I ran into an issue I didn't consider initially: upon failover we're umounting `/mnt/nfs/dumps` whenever the nfs client starts returning `ESTALE`. This works fine within the host, however containers that bind-mount the directory will not see the umount by default. Linux mount namespaces support the concept of propagating mount events for bind mounts, this feature is exposed as https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation `HostToContainer` by k8s and `rslave` mount option. The catch is that propagation works for subdirectories under the bind-mount, not for the bind-mount itself, in other words we need to mount a subdirectory of the actual mountpoint within the container. My first instinct was to bind-mount `/mnt/nfs` though I believe that's a non-starter because it also contains home/project mounts? I hope I'm wrong though. The alternative of course is to establish sth like `/mnt/nfs/dumps/mount` and then bind-mount `/mnt/nfs/dumps`. Reproducer to illustrate the behavior of parent directory vs exact bind mount setup ``` mkdir -p /root/proptest/nfs mount --bind /root/proptest /root/proptest mount --make-shared /root/proptest mount -t tmpfs nfs-sim /root/proptest/nfs echo "original content" > /root/proptest/nfs/marker.txt apt install containerd ctr image pull docker.io/library/alpine:latest ``` bind mount rslave the parent directory ``` ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propA sleep infinity ctr task exec --exec-id t1 propA cat /data/nfs/marker.txt # remount, NEW content will be shown umount /root/proptest/nfs mount -t tmpfs nfs-sim2 /root/proptest/nfs echo "NEW content after remount" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t2 propA cat /data/nfs/marker.txt ``` bind mount the exact mountpoint, content is not propagated ``` ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest/nfs,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propB sleep infinity ctr task exec --exec-id t3 propB cat /data/marker.txt # remount umount /root/proptest/nfs mount -t tmpfs nfs-sim3 /root/proptest/nfs echo "EVEN NEWER content" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t4 propB cat /data/marker.txt ```

While deploying `dumps-nfs.w.o` mounted to `/mnt/nfs/dumps` I ran into an issue I didn't consider initially: upon failover we're umounting `/mnt/nfs/dumps` whenever the nfs client starts returning `ESTALE`. This works fine within the host, however containers that bind-mount the directory will not see the umount by default. Linux mount namespaces support the concept of propagating mount events for bind mounts, this feature is exposed as https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation `HostToContainer` by k8s and `rslave` mount option. The catch is that propagation works for subdirectories under the bind-mount, not for the bind-mount itself, in other words we need to mount a subdirectory of the actual mountpoint within the container. My first instinct was to bind-mount `/mnt/nfs` though I believe that's a non-starter because it also contains home/project mounts? I hope I'm wrong though. The alternative of course is to establish sth like `/mnt/nfs/dumps/mount` and then bind-mount `/mnt/nfs/dumps`. Reproducer to illustrate the behavior of parent directory vs exact bind mount setup ``` mkdir -p /root/proptest/nfs mount --bind /root/proptest /root/proptest mount --make-shared /root/proptest mount -t tmpfs nfs-sim /root/proptest/nfs echo "original content" > /root/proptest/nfs/marker.txt apt install containerd ctr image pull docker.io/library/alpine:latest ``` bind mount rslave the parent directory ``` ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propA sleep infinity ctr task exec --exec-id t1 propA cat /data/nfs/marker.txt # remount, NEW content will be shown umount /root/proptest/nfs mount -t tmpfs nfs-sim2 /root/proptest/nfs echo "NEW content after remount" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t2 propA cat /data/nfs/marker.txt ``` bind mount the exact mountpoint, content is not propagated ``` ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest/nfs,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propB sleep infinity ctr task exec --exec-id t3 propB cat /data/marker.txt # remount umount /root/proptest/nfs mount -t tmpfs nfs-sim3 /root/proptest/nfs echo "EVEN NEWER content" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t4 propB cat /data/marker.txt ``` === deployment strategy * [ ] get new `volume-admission` with mountpropagation hosttocontainer and bind-mount `/mnt/nfs` * [ ] deploy to toolsbeta, enable dumps-nfs * [ ] restart all pods to pick up the changes * [ ] verify functionality * [ ] repeat for tools, possibly announcing the change

While deploying `dumps-nfs.w.o` mounted to `/mnt/nfs/dumps` I ran into an issue I didn't consider initially: upon failover we're umounting `/mnt/nfs/dumps` whenever the nfs client starts returning `ESTALE`. This works fine within the host, however containers that bind-mount the directory will not see the umount by default. Linux mount namespaces support the concept of propagating mount events for bind mounts, this feature is exposed as https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation `HostToContainer` by k8s and `rslave` mount option. The catch is that propagation works for subdirectories under the bind-mount, not for the bind-mount itself, in other words we need to mount a subdirectory of the actual mountpoint within the container. My first instinct was to bind-mount `/mnt/nfs` though I believe that's a non-starter because it also contains home/project mounts? I hope I'm wrong though. The alternative of course is to establish sth like `/mnt/nfs/dumps/mount` and then bind-mount `/mnt/nfs/dumps`. Reproducer to illustrate the behavior of parent directory vs exact bind mount setup ``` mkdir -p /root/proptest/nfs mount --bind /root/proptest /root/proptest mount --make-shared /root/proptest mount -t tmpfs nfs-sim /root/proptest/nfs echo "original content" > /root/proptest/nfs/marker.txt apt install containerd ctr image pull docker.io/library/alpine:latest ``` bind mount rslave the parent directory ``` ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propA sleep infinity ctr task exec --exec-id t1 propA cat /data/nfs/marker.txt # remount, NEW content will be shown umount /root/proptest/nfs mount -t tmpfs nfs-sim2 /root/proptest/nfs echo "NEW content after remount" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t2 propA cat /data/nfs/marker.txt ``` bind mount the exact mountpoint, content is not propagated ``` ctr run -d --snapshotter native \ --mount type=bind,src=/root/proptest/nfs,dst=/data,options=rbind:rslave \ docker.io/library/alpine:latest propB sleep infinity ctr task exec --exec-id t3 propB cat /data/marker.txt # remount umount /root/proptest/nfs mount -t tmpfs nfs-sim3 /root/proptest/nfs echo "EVEN NEWER content" > /root/proptest/nfs/marker.txt ctr task exec --exec-id t4 propB cat /data/marker.txt ``` === deployment strategy * [ ] get new `volume-admission` with mountpropagation hosttocontainer and bind-mount `/mnt/nfs` * [ ] deploy to toolsbeta, enable dumps-nfs * [ ] restart all pods to pick up the changes * [ ] verify functionality * [ ] repeat for tools, possibly announcing the change