Page MenuHomePhabricator

Make sure dumps-nfs mount/umount is propagated inside Toolforge containers
Closed, ResolvedPublic

Description

While deploying dumps-nfs.w.o mounted to /mnt/nfs/dumps I ran into an issue I didn't consider initially: upon failover we're umounting /mnt/nfs/dumps whenever the nfs client starts returning ESTALE. This works fine within the host, however containers that bind-mount the directory will not see the umount by default.

Linux mount namespaces support the concept of propagating mount events for bind mounts, this feature is exposed as https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation HostToContainer by k8s and rslave mount option.

The catch is that propagation works for subdirectories under the bind-mount, not for the bind-mount itself, in other words we need to mount a subdirectory of the actual mountpoint within the container. My first instinct was to bind-mount /mnt/nfs though I believe that's a non-starter because it also contains home/project mounts? I hope I'm wrong though.

The alternative of course is to establish sth like /mnt/nfs/dumps/mount and then bind-mount /mnt/nfs/dumps.

Reproducer to illustrate the behavior of parent directory vs exact bind mount

setup

mkdir -p /root/proptest/nfs
mount --bind /root/proptest /root/proptest
mount --make-shared /root/proptest

mount -t tmpfs nfs-sim /root/proptest/nfs
echo "original content" > /root/proptest/nfs/marker.txt

apt install containerd
ctr image pull docker.io/library/alpine:latest

bind mount rslave the parent directory

ctr run -d --snapshotter native \
  --mount type=bind,src=/root/proptest,dst=/data,options=rbind:rslave \
  docker.io/library/alpine:latest propA sleep infinity

ctr task exec --exec-id t1 propA cat /data/nfs/marker.txt

# remount, NEW content will be shown

umount /root/proptest/nfs
mount -t tmpfs nfs-sim2 /root/proptest/nfs
echo "NEW content after remount" > /root/proptest/nfs/marker.txt

ctr task exec --exec-id t2 propA cat /data/nfs/marker.txt

bind mount the exact mountpoint, content is not propagated

ctr run -d --snapshotter native \
  --mount type=bind,src=/root/proptest/nfs,dst=/data,options=rbind:rslave \
  docker.io/library/alpine:latest propB sleep infinity

ctr task exec --exec-id t3 propB cat /data/marker.txt

# remount
umount /root/proptest/nfs
mount -t tmpfs nfs-sim3 /root/proptest/nfs
echo "EVEN NEWER content" > /root/proptest/nfs/marker.txt

ctr task exec --exec-id t4 propB cat /data/marker.txt

deployment strategy

  • get new volume-admission with mountpropagation hosttocontainer and bind-mount /mnt/nfs
  • deploy to toolsbeta, enable dumps-nfs
  • restart all pods to pick up the changes
  • verify functionality
  • repeat for tools, possibly announcing the change

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
server: default to HostToContainer propagated mountsrepos/cloud/toolforge/volume-admission!51filippobug/T432325main
Customize query in GitLab

Event Timeline

My first instinct was to bind-mount /mnt/nfs though I believe that's a non-starter because it also contains home/project mounts?

This is true, though right now we either mount all of them, or none of them, so technically that would be ok. I think it's ok for now, when/if we want to start allowing mounting different ones independently (ex. expanding the values of --mount=all|none) we would be able to define a bit more clearly the structure that we want, like /mnt/nfs/public-data-services/{dumps,newthing,...} and /mnt/nfs/internal-data/{user_homes,tool_homes} or similar.

Good point @dcaro, ok thank you! So as far as Toolforge is concerned we can move to bind-mounting /mnt/nfs with mount propagation, I'll send out patches

fgiunchedi renamed this task from Make sure dumps-nfs mount/umount is propagated inside containers to Make sure dumps-nfs mount/umount is propagated inside Toolforge containers.Mon, Jul 20, 10:18 AM

group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1348

volume-admission: bump to 0.0.87-20260727092249-b2d76db8

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-27T09:29:35Z] <filippo@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T432325)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-27T09:32:29Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T432325)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-27T10:10:24Z] <filippo@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers (T432325)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-27T10:30:19Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers (T432325)

Mentioned in SAL (#wikimedia-cloud) [2026-07-27T11:14:43Z] <godog> set 'dumps_use_nfs_lb: true' for prefix-puppet toolsbeta-test-k8s-worker-nfs - T432325

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-27T11:37:00Z] <filippo@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T432325)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-27T11:39:58Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T432325)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-27T12:06:02Z] <filippo@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers (T432325)

Mentioned in SAL (#wikimedia-cloud) [2026-07-28T06:56:15Z] <godog> resume nfs workers roll restart - T432325

Mentioned in SAL (#wikimedia-cloud) [2026-07-29T08:56:37Z] <godog> enable 'dumps_use_nfs_lb: true' for bastion/static/mail prefixes - T432325

Mentioned in SAL (#wikimedia-cloud) [2026-07-29T09:47:59Z] <godog> enable 'dumps_use_nfs_lb: true' for nfs worker prefix - T432325

Mentioned in SAL (#wikimedia-cloud) [2026-07-29T09:54:33Z] <godog> test dumps-nfs failover - T432325

fgiunchedi claimed this task.

This is done -- toolforge now propagates /mnt/nfs mounts from the host to the containers and uses dumps-nfs.wikimedia.org load balanced service