Page MenuHomePhabricator

Soft mount option via hiera for NFS
Closed, ResolvedPublic


There's still quite a few projects that have NFS, many which use it for one of the following:

  1. Backups (on /data/project)
  2. Simply sharing files between different instances
  3. Actively reading / writing from it

For most of (1) and (2), we can soft mount them. This should allow the instances to continue to function despite not having NFS. This is a tracking ticket to look at each project, and soft mount NFS there when appropriate, which should be most cases.

Toolforge runs a database off NFS, so we are not moving that to soft mounts. In general soft mounting home directories can be kind of a bad idea as well. In a lot of cases, though, writes are not constant and a single file corrupted isn't the end of the world, and a hiera option will make those use cases more resilient.

Event Timeline

chasemp triaged this task as Medium priority.Apr 4 2016, 2:25 PM
Bstorm renamed this task from Soft mount NFS for almost (all) projects that still have NFS to Soft mount option via hiera for NFS.Jun 11 2020, 10:25 PM
Bstorm updated the task description. (Show Details)

Providing a way to soft mount some connections should help a lot now that we know why the soft-mounted dumps servers caused infinitely climbing load on the Kubernetes cluster when they were down ( see also T247455: Dumps not accessible from container pods).

So cool thing: effectively introduced exactly the option I want here. It seems like a dangerous thing to soft mount home directories, but that could be done with a similar opt-in sort of config. Everything else is already soft mounted. I'm going to test the setting in a place or two and then put up a patch to allow /home opt-in as well for folks who are brave.

Mentioned in SAL (#wikimedia-cloud) [2020-06-12T19:41:49Z] <bstorm_> set profile::wmcs::nfsclient::mode: soft on toolsbeta-workflow-test T127559

That worked great! On specific prefixes in horizon, that would even work for some tools servers. Ok, making a patch.

Change 605310 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: allow opt-in soft mounting wherever folks want to try it

Change 605310 merged by Bstorm:
[operations/puppet@production] cloud nfs: allow opt-in soft mounting wherever folks want to try it