Can Cinder be used to move data from one machine to another? Moreover, would it be possible to get additional storage for individual machines, as the default HDD size is limited? Also, we would need to move the current data in the HDFS (mostly persistent settings) to another location.

Can Cinder be used to move data from one machine to another?

A cinder volume can be detached from one VM and attached to another. Only one VM can be attached a time, though. So it's good for e.g. persisting data over a VM rebuild or upgrade, but doesn't provide the multi-host shared filesystem that NFS does.

Moreover, would it be possible to get additional storage for individual machines, as the default HDD size is limited?

Now that Cinder is available, all new VMs have only a 20Gb OS drive -- all other storage is managed via Cinder. We can definitely increase Cinder quotas for your project, especially if that quota is used to replace an NFS use case.

OK. If we can get additional temporary storage, that would be as good as NFS.
We will be able to work with persistent storage and share the config via a private git repo (somehow).

Previously we were linking /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/ from within our project NFS to filter out pages that contain math in all languages and build individual wikis to test new math features. However, since this resource seems to be gone, we will need to implement workflow anyhow. Probably, we can download one dump at a time via HTTP and delete it afterward. That will still be slower, but not waste too much disk space.

Ok... two things. First of all -- I meant this ticket to only be about your shared /data/project shared mount, which is project-specific. I expect your NFS mount of the dumps directories to remain.

Secondly...

In T301300#7701368, @Physikerwelt wrote:

Previously we were linking /mnt/nfs/dumps-labstore1007.wikimedia.org/xmldatadumps/public/

I'm confused by this. Your VMs (e.g. math19) definitely have dumps mounted at /mnt/nfs/dumps-labstore1007.wikimedia.org and /mnt/nfs/dumps-labstore1006.wikimedia.org and all the data is there. You wouldn't be able to see the /xmldatadumps/public/ path because that is the path (on the server) that is mounted on your host as /mnt/nfs/dumps-labstore1007.wikimedia.org/

Does that make sense? Are you still unable to find dumps that you need?

I see this was a different problem. The dumps from that time in 2019 were deleted. The vision was that one would be able to reproduce the exact same experiment in the future and obtain the same result. At least the latest dumps are there, so if we repeat the experiment with new data, the same script will work but produce different results. That being said, to make the experiment reproducible, we will need to store the data anyhow.

After all, I think NFS is not needed and using git and rsync are viable alternatives to share data between VMs. What is the timeline for shutting off NFS?

In T301300#7717201, @Physikerwelt wrote:

After all, I think NFS is not needed and using git and rsync are viable alternatives to share data between VMs. What is the timeline for shutting off NFS?

I'm moving the /data/project and $home mounts off of the central NFS server and onto project-local VMs. If you need some time to re-engineer I can just move your project to a local server for now and you can just let me know when you're ready to have that NFS setup removed.

On the other hand, if you think you can move off NFS in the next week or two then I'll wait.

But, to be clear -- we aren't dropping support for NFS, just trying to avoid using it where we don't have to. If it works for you and solves a real problem then you can keep it! There'll just be a bit of downtime and some reboots while I move the storage.

@Physikerwelt Have you formed an opinion about this? I'm happy leaving things as they are if that suits you.

If we can keep it for now that would be awesome. We thus won't use it for new developments.

@Physikerwelt I'm going to transfer this project to a project-local NFS server sometime soon. The easiest way to do that is to shut down all hosts in the project while moving; is there a better or worse time for that to happen or should I just do it whenever?

Now (within the next month) there is an excellent time.

Sorry, I did not really understand what Cinder is and if it would be better (for the long-term maintenance effort) if we moved there?

Cinder works great and anytime you need additional storage attached to /one VM at a time/ Cinder is the right solution.

If, on the other hand, you need to share a file system between multiple VMs, NFS is still your solution.

LMK which of those is your use case :)

Thank you. This means we need NFS.

Got it. I have some free time after lunch, so get ready for some downtime :)

I've moved you to a project-local nfs server which you can now see in horizon, named 'math-nfs-1' -- both your hosts are up and running and the file situation should be just as it was.

You should not need to do any specific maintenance on that new VM but be aware that it's now serving all of your shared files so is best left up :)

Please re-open this ticket if you run into anything surprising.