Page MenuHomePhabricator

Purge and monitor old metadata for the mariadb backups database
Open, LowPublic


So far, metadata for backups is stored on the zarcillo database indefinitely. Maintaining, e.g. one year of backlog of backups is doable (we will only have around 50 * number of sections * number of datacenters). However, backups are sent to long term storage (bacula) after a week, and removed from the local storage after 3 weeks. Those are not really needed, and may slow down the operations a lot if a lot of data is stored (depending on the analytics needed).

Consider purging or setting up a "deleted" state for older backups, either on purge ran (rotation), or out of band with a cron job.

See also if something about the metadata lifecycle has to be improved somehow (e.g. tracking while on bacula storage only, etc.)

Edit: Additionally, monitor long-running ongoing backups and mark them as failed (for example, after 24 hours), and/or alert on those.

Event Timeline

jcrespo triaged this task as Medium priority.Sep 27 2018, 2:36 PM
jcrespo created this task.
jcrespo renamed this task from Purge old metadata for the mariadb backups database to Purge and monitor old metadata for the mariadb backups database.Mar 21 2019, 10:39 AM
jcrespo updated the task description. (Show Details)
jcrespo removed a subscriber: Banyek.
jcrespo lowered the priority of this task from Medium to Low.Sep 9 2020, 3:13 PM

We can keep for now the history of all backups performed.