Page MenuHomePhabricator

Local private files on deployment host should be backed up somewhere
Open, MediumPublic

Details

Reference
bz67818

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:36 AM
bzimport set Reference to bz67818.

14:53 < Reedy> Needs backup/remote repo (and backup of that)

Maybe put it the private ops repo we have for puppet configs?

(In reply to Krinkle from comment #2)

Maybe put it the private ops repo we have for puppet configs?

Then no one who can deploy, who may need to change PrivateSettings (most likely, adding one), will be able to do so. Presumably that then also depends on puppet ensuring it's in place on tin, to then be synced out. Unless it goes somewhere else on the apache (et al) file systems, and we include/symlink it from there...

I've created a private repo in /a/common/private, but it needs a remote to push somewhere (so it can be backed up), but obviously not gerrit. That's an aside. Not synced to production

AdminSettings.php merged into PrivateSettings.php in the 2nd commit to the repo.

private added to .gitignore in https://gerrit.wikimedia.org/r/145403

PrivateSettings.php and AdminSettings.php removed from .gitignore, and replaced with symlinks to ../private/PrivateSettings.php in https://gerrit.wikimedia.org/r/145404

Need to do something similar on beta for testing (and so it doesn't break it)

Andrew added a subscriber: Andrew.

Many of these settings come from puppet, so the raw file can't be in mw repo; it's already puppetized as intended.

Wait, I'm totally wrong, confusing this with something else

Reedy removed Reedy as the assignee of this task.Sep 27 2015, 2:02 PM
Reedy added a subscriber: Reedy.
Krenair renamed this task from PrivateSettings.php should be in a repo to Private repository should be pushed to somewhere remote.Sep 27 2015, 2:05 PM
Krenair updated the task description. (Show Details)
Krenair set Security to None.
Krenair added a subscriber: Krenair.
Krinkle renamed this task from Private repository should be pushed to somewhere remote to Local private files on deployment host should be in a repo somewhere.Aug 7 2018, 8:56 PM
Krinkle renamed this task from Local private files on deployment host should be in a repo somewhere to Local private files on deployment host should be backed up somewhere.Aug 7 2018, 8:59 PM

The issue of having both AdminSettings and PrivateSettings was fixed (T69820, removed AdminSettings).

The issue of not having a backup of recent versions of PrivateSettings still remains, narrowing this task to be about that.

Is PrivateSettings.php a single file that exists on the deployment host and that is shared between everyone with deploy access? Or does everyone have their own?

Is PrivateSettings.php a single file that exists on the deployment host and that is shared between everyone with deploy access? Or does everyone have their own?

It's got a list of passwords and stuff used in wmf-config. So everyone "shares" it

The folder/repo is also used for some spam prevention things we don't want in public

Krinkle added a subscriber: jcrespo.

I believe this has since been done, per T125527: Backup all of /srv on mira and/or tin (deployment servers).

@jcrespo Can you confirm that /srv/mediawiki-staging/private/* is covered by baculate (eg. not somehow excluded from /srv/).

This is the contents on backups for deploy1002 as of yesterday (under NDA, because I don't know if the file names are sensitive):

(Go to line 279)
{P17346}

And this is the contents for deploy2002 (also NDA):

(Go to line 283)
{P17347}

So I consider it covered.

@Krinkle before closing this, could you ask me to restore a file at some point in time (within the last 2 months, and we don't have to replace existing files, we can recover to a separate path) to make sure the backups are working as expected- we should regularly test recoveries to make sure they are working, but "testing" tasks will vary from service to service.

Waiting for an answer, even if it is "no". :-)

@jcrespo OK, could you restore /srv/mediawiki-staging/private/PrivateSettings.php from ~4 weeks ago to my home directory?

Done, the exact file is at deploy1002:/home/krinkle/restore/srv/mediawiki-staging/private/PrivateSettings.php and its mod time is 2021-05-26 17:14:22 You can diff it against current one and see if it corresponds to the status at 2021-09-16 04:13:11, the moment the backup took place (4 weeks earlier- we have daily backups).

I didn't modify the original file permissions.

Please check the file is as expected, otherwise please communicate any issue so I can solve it.

If everything looks right, please delete it from your home (you should have the permissions, otherwise ping any root) and we can resolve this ticket :-). Thanks for testing the backups, they are forgotten sometimes but they help making sure they are ready at an unexpected time!

Log of commands for the record:

ssh backup1001.eqiad.wmnet
$ sudo bconsole
*restore
*6
*2021-09-16 08:41:00
*75
*2
*cd /srv/mediawiki-staging/private/
*ls
*mark PrivateSettings.php
*done
*mod
*9
*/home/krinkle/restore
*yes