Soft mount option via hiera for NFS
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	yuvipanda
	Feb 19 2016, 10:23 PM

Description

There's still quite a few projects that have NFS, many which use it for one of the following:

Backups (on /data/project)
Simply sharing files between different instances
Actively reading / writing from it

For most of (1) and (2), we can soft mount them. This should allow the instances to continue to function despite not having NFS. This is a tracking ticket to look at each project, and soft mount NFS there when appropriate, which should be most cases.

Toolforge runs a database off NFS, so we are not moving that to soft mounts. In general soft mounting home directories can be kind of a bad idea as well. In a lot of cases, though, writes are not constant and a single file corrupted isn't the end of the world, and a hiera option will make those use cases more resilient.

Details

	Subject	Repo	Branch	Lines +/-
	cloud nfs: allow opt-in soft mounting wherever folks want to try it	operations/puppet	production	+12 -4

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• Bstorm	T126083 overhaul labstore setup [tracking]
		Resolved		• Bstorm	T127559 Soft mount option via hiera for NFS

Event Timeline

yuvipanda created this task.Feb 19 2016, 10:23 PM

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 19 2016, 10:23 PM

yuvipanda added a parent task: T126083: overhaul labstore setup [tracking].Feb 19 2016, 10:24 PM

zhuyifei1999 subscribed.Mar 24 2016, 3:38 AM

• chasemp triaged this task as Medium priority.Apr 4 2016, 2:25 PM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:44 PM

• bd808 edited projects, added Data-Services; removed Cloud-Services.Jul 23 2017, 11:38 PM

• bd808 moved this task from Backlog to Shared Storage on the Data-Services board.Jul 24 2017, 12:40 AM

• GTirloni added a project: cloud-services-team (Kanban).Mar 23 2019, 9:21 PM

• Bstorm renamed this task from Soft mount NFS for almost (all) projects that still have NFS to Soft mount option via hiera for NFS.Jun 11 2020, 10:25 PM

• Bstorm updated the task description. (Show Details)

Providing a way to soft mount some connections should help a lot now that we know why the soft-mounted dumps servers caused infinitely climbing load on the Kubernetes cluster when they were down ( see also T247455: Dumps not accessible from container pods).

So cool thing: https://gerrit.wikimedia.org/r/c/operations/puppet/+/329711 effectively introduced exactly the option I want here. It seems like a dangerous thing to soft mount home directories, but that could be done with a similar opt-in sort of config. Everything else is already soft mounted. I'm going to test the setting in a place or two and then put up a patch to allow /home opt-in as well for folks who are brave.

Mentioned in SAL (#wikimedia-cloud) [2020-06-12T19:41:49Z] <bstorm_> set profile::wmcs::nfsclient::mode: soft on toolsbeta-workflow-test T127559

That worked great! On specific prefixes in horizon, that would even work for some tools servers. Ok, making a patch.

Change 605310 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: allow opt-in soft mounting wherever folks want to try it

https://gerrit.wikimedia.org/r/605310

gerritbot added a project: Patch-For-Review.Jun 12 2020, 7:56 PM

• Bstorm claimed this task.Jun 16 2020, 6:11 PM

• Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 605310 merged by Bstorm:
[operations/puppet@production] cloud nfs: allow opt-in soft mounting wherever folks want to try it

https://gerrit.wikimedia.org/r/605310

Mentioned in SAL (#wikimedia-cloud) [2020-06-16T21:33:04Z] <bstorm_> setting "profile::wmcs::nfsclient::mode: soft" on the project T102369 T127559