(In reply to Krinkle from comment #2)
Maybe put it the private ops repo we have for puppet configs?
Then no one who can deploy, who may need to change PrivateSettings (most likely, adding one), will be able to do so. Presumably that then also depends on puppet ensuring it's in place on tin, to then be synced out. Unless it goes somewhere else on the apache (et al) file systems, and we include/symlink it from there...
I've created a private repo in /a/common/private, but it needs a remote to push somewhere (so it can be backed up), but obviously not gerrit. That's an aside. Not synced to production
AdminSettings.php merged into PrivateSettings.php in the 2nd commit to the repo.
private added to .gitignore in https://gerrit.wikimedia.org/r/145403
PrivateSettings.php and AdminSettings.php removed from .gitignore, and replaced with symlinks to ../private/PrivateSettings.php in https://gerrit.wikimedia.org/r/145404
Need to do something similar on beta for testing (and so it doesn't break it)
I believe this has since been done, per T125527: Backup all of /srv on mira and/or tin (deployment servers).
@jcrespo Can you confirm that /srv/mediawiki-staging/private/* is covered by baculate (eg. not somehow excluded from /srv/).
This is the contents on backups for deploy1002 as of yesterday (under NDA, because I don't know if the file names are sensitive):
(Go to line 279)
And this is the contents for deploy2002 (also NDA):
(Go to line 283)
So I consider it covered.
@Krinkle before closing this, could you ask me to restore a file at some point in time (within the last 2 months, and we don't have to replace existing files, we can recover to a separate path) to make sure the backups are working as expected- we should regularly test recoveries to make sure they are working, but "testing" tasks will vary from service to service.
Done, the exact file is at deploy1002:/home/krinkle/restore/srv/mediawiki-staging/private/PrivateSettings.php and its mod time is 2021-05-26 17:14:22 You can diff it against current one and see if it corresponds to the status at 2021-09-16 04:13:11, the moment the backup took place (4 weeks earlier- we have daily backups).
I didn't modify the original file permissions.
Please check the file is as expected, otherwise please communicate any issue so I can solve it.
If everything looks right, please delete it from your home (you should have the permissions, otherwise ping any root) and we can resolve this ticket :-). Thanks for testing the backups, they are forgotten sometimes but they help making sure they are ready at an unexpected time!
Log of commands for the record:
ssh backup1001.eqiad.wmnet $ sudo bconsole *restore *6 *2021-09-16 08:41:00 *75 *2 *cd /srv/mediawiki-staging/private/ *ls *mark PrivateSettings.php *done *mod *9 */home/krinkle/restore *yes