Page MenuHomePhabricator

print a list of backed up directories in the MOTD of production servers
Closed, ResolvedPublic

Description

When logging in, print the list of directories/files backed up from a particular server so it is immediately obvious when logging in.

Event Timeline

LSobanski triaged this task as Medium priority.Jan 22 2021, 11:36 AM
LSobanski moved this task from Triage to Refine on the Data-Persistence-Backup board.
LSobanski subscribed.

Sounds like a good idea. Is this to address a specific concern that came up? One thing that comes to mind is the amount, readability and relevance of the information, e.g. maybe it would make sense to only limit this to directories the account has write access to (tricky for root)? Some UX testing would be nice :)

Is this to address a specific concern that came up?

^@mark

It's purely an idea I've had for a long time, to make it immediately obvious to anyone logging in what is backed up, and what isn't. That should help to:

  • help shell users determine where they should and should not store information they need to keep, and
  • help identify/remind where we have gaps in our backup coverage. More readily available information than "somewhere in an inventory or in Puppet".

For the latter reason, it may still be useful for users to see the full list, even if they may not have write access to part of it.

2 notices:

  • This could only cover direct bacula backups (not things that are indirectly backed up, like puppet or gerrit repos, or database and media backups, not covered by bacula)
  • 'More readily available information than "somewhere in an inventory or in Puppet"'. We started trying to make things more user friendly with the list at https://grafana.wikimedia.org/d/413r2vbWk/bacula , although I agree that is not enough

Apparently, there is the following code on backup::set:

$motd_content = "#!/bin/sh\necho \"Backed up on this host: ${name}\""
        @motd::script { "backups-${name}":
            ensure   => present,
            priority => 6,
            content  => $motd_content,
            tag      => 'backup-motd',
        }
}

In other words, this was prepared to do exactly that since 2014. But AFAICS, this doesn't take effect. Will research why and enable it if reasonable.

Change 659056 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Fix bug on realizing backup sets on motd

https://gerrit.wikimedia.org/r/659056

Change 659056 merged by Jcrespo:
[operations/puppet@production] bacula: Fix bug on realizing backup sets on motd

https://gerrit.wikimedia.org/r/659056

$ ssh apt1001.wikimedia.org 
Linux apt1001 XXXXX-amd64 #1 SMP Debian XXXXX (XXXXX) x86_64
Debian GNU/Linux 10 (buster)
Backed up on this host: roothome
Backed up on this host: srv-autoinstall
Backed up on this host: srv-wikimedia
The last Puppet run was at Thu Jan 28 12:00:19 UTC 2021 (0 minutes ago). 
Last puppet commit: (88a98f1174) Jcrespo - bacula: Fix bug on realizing backup sets on motd
Debian GNU/Linux 10 auto-installed on XXXXX.
Last login: XXXXX from XXXXX

@mark is this enough? We could print individual defined files and directories, but it would get a bit messy (too much output on startup if config too long, and puppet code a bit more complex than necessary, potentially making adding new backup datasets more complex).

$ ssh gerrit1001.wikimedia.org 
...
Backed up on this host: gerrit-repo-data
...
jcrespo closed this task as Resolved.EditedJan 29 2021, 11:47 AM
jcrespo claimed this task.

I consider this solved, as I got no negative feedback on it. We can reevaluate printing more detailed stats once we refactor puppet code for bacula more deeply.