Page MenuHomePhabricator

Restore ~tjones/reindex directory from mwmaint1002
Closed, ResolvedPublic

Description

When mwmaint1002.eqiad.wmnet was reimaged (at 2021-07-13 11:13, according to @RLazarus, who helped me out on IRC), I lost some files and data in the directory ~tjones/reindex.

I'd really appreciate it if that directory could be restored, to mwmaint1002 or mwmaint2002 or anywhere else I can get to it—whatever makes sense.

Reuven also discovered that...

... restoring it is a little trickier in this case, since the host was reimaged -- normally the backups are decrypted with a host key, but since that key is gone, we'll decrypt using the master key instead ... that process is a little more involved ...

Hence, this ticket! Thanks!

Event Timeline

My naive attempt at https://wikitech.wikimedia.org/wiki/Bacula#Restore_(aka_Panic_mode) went fine until the decryption phase, at which point "Error: Missing private key required to decrypt encrypted backup data." showed up in the bconsole messages, as expected in this case since the machine was reimaged.

The instructions at #Restore_from_a_non-existent_host_(missing_private_key) look clear, but I didn't want to try them for the first time on my own, with the data persistence team already offline for the weekend. :) Thanks in advance for picking up where I left off.

Any idea when someone might have time to look at this?

I'm trying to avoid having to recreate code that I had on mwmaint1002, but I have another ticket that's blocked until either I do that, or I can get access to the backup.

Sorry to be a pest.

We plan to work on this today. Sadly, for some reason, phabricator didn't send me any email about this issue until the end of my day yesterday, so I had to get pinged by rzl directly. Unless something bad happens, it should be ready at the start of your day.

@TJones I've restored your old home folder onto mwmaint1002:/home/tjones/backup-restore-2021-07-13--05-05-51

It should have the same access permissions as the old one, but please comment if you have any issue accessing them. We have restored the backup taken on the 13th of July, previous to the wipe.

The files being lost is, of course, not your fault, but I would like to take the opportunity to encourage, if there is important functionality there for you, to send a patch to the operations/puppet repository or your own separate one (that we can tell puppet to checkout on mw maintenance hosts)-- I will be glad to assist you with such process. My worry is that in the future the hw fails, it could lose your changes (while the repos have both very frequent backups AND redundancy). My understanding is that, except for very specific hosts (people.wikimedia.org, maybe others), homes can be lost at any time, mostly for privacy reasons.

Thanks, @jcrespo! It looks like everything I need is there.