Page MenuHomePhabricator
Create Task
Maniphest T432583

Make sure dumps-nfs mounts are propagated inside PAWS containers
Open, Needs TriagePublic
Actions

Description

This is similar to T432325: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers except that it covers PAWS as a standalone/magnum cluster. We'll have to make sure of the following:

  • /mnt/nfs/dumps is auto-mounted to dumps-nfs.w.o
  • when failover/ESTALE happens, there's a "sitter" process to umount. i.e. similar to what puppet does
  • mounts are propagated inside containers, i.e. /mnt/nfs is bindmounted + rslave

In terms of implementation I am thinking the following:

  • A maintain-dumps-nfs script that periodically will: check the host if nfs mounts are mounted, if not mount them. if they are mounted and not healthy (ESTALE) then umount and re-mount
  • Said script will run inside a container image via a privileged daemonset on all non-control hosts

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Mon, Jul 20, 10:22 AM
fgiunchedi mentioned this in T432587: Migrate to dumps-nfs.w.o in Cloud VPS.Mon, Jul 20, 10:32 AM
fgiunchedi updated the task description. (Show Details)Wed, Jul 22, 8:07 AM
Stashbot added a comment.Tue, Jul 28, 7:15 AM

Mentioned in SAL (#wikimedia-cloud) [2026-07-28T07:15:49Z] <godog> set up clouddumps nfs audit via network capture - T432583

fgiunchedi added a comment.Thu, Aug 13, 7:59 AM

I realized github pull requests won't show up here. I got these out:

https://github.com/toolforge/paws/pull/521
https://github.com/toolforge/paws/pull/522
https://github.com/toolforge/paws/pull/523

To implement the above (i.e. a maintain-dumps-nfs daemonset to do the mount handling)

The testing rig I'm using for "local" paws testing is a bit janky but it works:

ssh pontoon-demo-tf-services-01.testlabs.eqiad1.wikimedia.cloud -L8000:192.168.49.2:80
(pontoon is not involved I just had that VM available)

then locally on your host do: echo '127.0.0.1 hub.paws.local public.hub.paws.local' >> /etc/hosts

and then you have the hub at http://hub.paws.local:8000/  (http not https)

once you have the paws repo cloned on the remote host and with the commits/branche you want to test, deploy is:

helm -n paws-dev upgrade --install dev paws/ --timeout=50m  --set maintainDumpsNfs.enabled=true
Restricted Repository Identity mentioned this in rPAWS42a9f379ee51: Mount dumps only in singleuser containers (#521).Thu, Aug 13, 11:24 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits