Page MenuHomePhabricator

move math-nfs-1 to new instance
Closed, ResolvedPublic

Description

I have created a Debian 13 version math-nfs-2, copied the puppet classes, and reattached the volume. However, after restarting the other lab instances, they don't seem to find the respective nfs mounts. They create empty directories. How do they figure out that math-nfs-2 is the new host serving NFS?

Note, without a restart, the other instances had a broken link to the mount point

physikerwelt@math-docker-2:/mnt/nfs$ ls -lah
ls: cannot access 'labstore-secondary-project': Stale file handle
total 84K
drwxr-xr-x    6 root root 4.0K Jan 11  2024 .
drwxr-xr-x    3 root root 4.0K Jan 11  2024 ..
drwxr-xr-x 1102  400  400  36K Jun 17 19:45 dumps-clouddumps1001.wikimedia.org
drwxr-xr-x 1102  400  400  36K Jun 17 19:45 dumps-clouddumps1002.wikimedia.org
d?????????    ? ?    ?       ?            ? labstore-secondary-project
drwxrwxrwt   22 root root 4.0K Jun  9 13:22 secondary-scratch
physikerwelt@math-docker-2:/mnt/nfs$

Event Timeline

The documentation at https://wikitech.wikimedia.org/wiki/Help:Shared_storage#/data/project does not mention that an NFS server needs to be configured.

You need to update the dns record for math-nfs.svc.math.eqiad1.wikimedia.cloud to the new instance, via https://horizon.wikimedia.org/ngdetails/OS::Designate::Zone/8fb8cf01-ac22-4fad-9495-7d6135574fa1

I'm not sure you'd be able to change the zone yourself, if that's not the case then please let us know (cfr https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Runbooks/Create_an_NFS_server#NFS_service_failover)

Physikerwelt claimed this task.

@fgiunchedi thank you so much

I made it. It worked almost without problems. Only I had to manually edit /etc/fstab to mount the second disk. I was hoping that this would already be taken care of by the puppet config..

@fgiunchedi thank you so much

I made it. It worked almost without problems. Only I had to manually edit /etc/fstab to mount the second disk. I was hoping that this would already be taken care of by the puppet config..

Interesting, IIRC fstab editing is taken care by wmcs-prepare-cinder-volume which in turn is run by the cookbooks here https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Runbooks/Create_an_NFS_server . I'm not sure if we have an end to end guide for users on how to self-service migrate their nfs servers. Maybe @Andrew knows

wmcs-prepare-cinder-volume

It also handles the formatting, so I was afraid that it might reformat the existing disk.

The puppet config links above says

profile::wmcs::nfs::standalone::cinder_attached: true

so maybe there is a step missing that configures the local mount.

However, maybe this is not a problem that many people might face. That's why I closed the ticket so more important tasks can be worked on. Thank you for your help.

I'm not sure if we have an end to end guide for users on how to self-service migrate their nfs servers. Maybe @Andrew knows

Typically I've maintained the NFS servers, as a holdover from when there was one monolithic server provided to all projects. I created T429793 for a start.