After some previous discussion, I have been exploring a bit all suggestions made there and, finally, I have written some notes with three different options to implement the disaster recovery plan for MPIC: https://docs.google.com/document/d/11exlY2vFBKpqUTKU_dwQdm88-oJn-94icM041kOe3-8

I have added my own considerations as "Advantages/Disadvantages" but it's fully open for discussion, changes, comments and everything you consider. In fact I have already added a couple of comments about a couple of things are not clear for me

I'm pretty sure I'll be wrong about some assumptions and details but I hope it can be, at least, a good starting point to reach the best way to do it.

@BTullis @phuedx Could you take a look at this?

@Sfaci: I took a look. Thanks for detailing the options that we have. Do you have any preference? Mine would be to stay close to what other teams are doing with similar systems but also to try to keep it as simple as possible.

@phuedx Fully agree! I think we shouldn't reinvent anything and, at the same time, simplicity is welcome. My dilemma here is that I think the simplest solution is the most custom/unusual (Option 3: embed a scheduled job within MPIC) and the Option 1 (Airflow DAG) seems to be usual but it would take more time to do it (at least from my experience). I think that needing to format the data according to the stream configuration format makes this a bit particular. So far my preference was Option 3 because of its simplicity but maybe it breaks with the usual way of doing things (although, to be honest, I'm not pretty sure what this usual way is regarding similar applications)

While writing these words a new idea has come to my mind, what about writing only a script we have to run manually? I mean, Bacula is already there backing up our MariaDB database weekly (is that enough?) so we already have a weekly backup according to the usual way things are done here. The only we would need is something to prepare that data to the right format every time we need to restore the backup. We could run the script manually when needed using the last backup. That should be pretty immediate so we wouldn't add any significant delay. And restoring the configuration would be already a manual step. Could that be acceptable? I don't like manual steps and I'd prefer to fully automate the process, but we take advantage of the current backup system, it's simple and we always could improve it scheduling the script at some point.

Thanks @BTullis for your feedback. It's really interesting and valuable. it's helping a lot to decide the right option here. So, according to what we know so far (https://docs.google.com/document/d/11exlY2vFBKpqUTKU_dwQdm88-oJn-94icM041kOe3-8), we could say that:

• Option 1 (Yarn script + Airflow DAG + Bacula) could be discarded because its complexity
• Option 2 (K8s CronJob + Bacula) seems doable and also simple. And there is no limitations regarding the script we can write. We could use nodejs, for example
• Option 3 (Background job embedded with MPIC) seems to be the simplest solution that meets all our needs; simple, automatic and we can configure periodicity.
• Option 4 (Manual script) It's even simpler than the previous one but we must run manually in case of disaster. With a tiny extra effort we could schedule the task using Option 3, for example
• Option 5 (scheduled task + S3 bucket): Seems great but more complex than others. We could discard it because of that and we always can improve the option we choose to include a S3 backup later

Having said that, I would say that our best choice right now is Option 3 using node-cron which is a nodejs module that works as a task scheduler. That way the schedule task will be part of MPIC and run within the same pod and configuration In addition to that this solution could be improved later if needed to include some other features we have mentioned in other options (S3, a separate pod with CronJob as a nodejs script, using timestamps to have multiple point-in-time recoveries, . . .)

It's decided, Option 3 it is.