Such as mysql backups, files processed, etc.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | yuvipanda | T105720 Labs team reliability goal for Q1 2015/16 | |||
| Declined | None | T104193 Find replacements for various things that people were using NFS for but should not have been (Tracking) | |||
| Duplicate | None | T104206 Provide a simple way to backup arbitrary files from instances |
Event Timeline
Comment Actions
Let's start by evaluating a central bacula server as we have in production that will be able to backup files from labs VMs. That way we will build on established knowledge and move fast. There are various technicalities to solve like production using exported resources and we can't in labs, shared secret population, usage or not of TLS and so on.
We have some decisions like retention policy (how long back in time will our backups go, how often we will be recreating them from scratch, and so on.
Later on, we will need some disk space estimations and procure some hardware.
Comment Actions
So ideally the workflow I'd like people to have:
- Specify filesets in hiera (so can do it per host or per-project)
- Specify retention strategy (from a few options we give them)
- Restores would go through phabricator
This would allow me to remove NFS from a few proejcts, and feel better about a lot more projects :)
Comment Actions
Agreed. Where however this will be defined in is a good question. The per project hiera sounds good enough for the users to configure but it is not so easily fetchable from the director/storage daemon's POVs. In production we solve that by having a single administrative domain and use exported resources but we clearly can't use that in the multi-project environments labs is.
- Specify retention strategy (from a few options we give them)
Nope. Let's not. Same retention policy for everything/everyone. There are anyway multiple levels of retention policies and I am pretty sure all most people will be confused by how they work. For example, per client retention policies are NOT guarantees the data will stay that long around. They more like guarantees the data is indexed. And the per volume retention policies can not be set per client.
- Restores would go through phabricator
Agreed.
This would allow me to remove NFS from a few proejcts, and feel better about a lot more projects :)
Comment Actions
Just +1ing this.
I'm struggling to implement a robust backup strategy for ORES and Wikilabels right now. Because I don't want the instability of NFS, I'm currently SCPing copies of my databases to my home server.
Comment Actions
how would you feel about us providing space for scp/rsync instead of NFS for you to back things up to?
simple quota'd persistent space