Page MenuHomePhabricator

Add step to rsync home dirs on mwmaint hosts before DC switchover
Closed, DeclinedPublic

Description

For better or worse, people tend to have various adhoc, unpuppetized scripts in their mwmaint home directories. When a new mwmaint server is set up, we typically rsync home dirs over, but that's a one time operation. It would be smoother if we just copied everything over before or right after the switchover process so aside from the hostname change, everything is basically the same.

Even though I wrote "before" in the description, I think it could happen right afterwards too, as a 08 or 09 cookbook.

Event Timeline

I personally disagree, untracked and unreviewed scripts should not run against production IMHO. And we shouldn't encourage and simplify this behaviour but instead make easier to have all scripts that will run against production tracked, reviewed and deployed in a standard way.

I personally disagree, untracked and unreviewed scripts should not run against production IMHO. And we shouldn't encourage and simplify this behaviour but instead make easier to have all scripts that will run against production tracked, reviewed and deployed in a standard way.

While I agree with you, that's not the current reality. In any case it's not just scripts, it's also log files, some data or query results for various investigations/projects - and that stuff can't really be puppetized.

While I agree with you, that's not the current reality. In any case it's not just scripts, it's also log files, some data or query results for various investigations/projects - and that stuff can't really be puppetized.

Sure but all those other files can still be accessed wherever they are, the users can still access any mwmaint host at any time, doesn't matter if it's the active one or not.
In addition mwmaint1002 was recently reimaged in T267607 and as far as I can tell the homes were not preserved / copied over (please correct me if I'm wrong). Looks like the opposite approach regarding how important is that data compared to the one outlined here.

I've also some technical questions on how you were envisioning this sync:

  • unidirectional or bi-directional?
  • with the objective of making the home directory in both servers identical or syncing hostA's /home/foo into hostsB's /home/foo/hostA/ (or similar)?
  • with or without rsync's `--delete` (and related) options?
  • the current size of /home in both servers is really different, and it seems that they can be large: 22MB on mwmaint1002 (due to recent reimage) vs 48GB on mwmaint2002.
  • the two hosts have very different disk size (102G on mwmaint1002 vs 539G on mwmaint2002), so the sync might fill up completely the disk of the smaller one

I'm also worried that a one time sync during the switchover might give the wrong impression that the data is kept in sync in a more continuous way between the two hosts and people might modify a script in one host and then run an outdated version of it on the other host.

Those are just my concerns/doubts, I'll leave it to you and your team to decide what's best.

akosiaris subscribed.

I am closing as Declined. We 've taken no action in 1.5 year. Also, @Volans is correct that there is quite a bit of complexity in there regarding how to implement this with apparently small benefits (some log files, some data or results that are still accessible, albeit under a different name). Furthermore, the future of mwmaint hosts in the mw-on-k8s world appears to be that they will be removed. While discussions are ongoing, I 'd say that devoting effort to solve this and then scrap the hosts altogether isn't a good use of time. Hence, declined and we can always re-open if this becomes important somehow again.