- Documentation, documentation, documentation
- More flexibility: if possible per-table CSVs
- More flexibility: Physical backups
- Better recovery documentation: "one line to recover" (there is now a recover_section.py)
- Faster point in time recovery/premade tools
- Better compression
- Prepare based on name of the backup, not just the section
- Option optimization (e.g. double the use_memory)
- More detailed health checks of backups (size, failures, objects, ...). E.g. check size is within a percentage of the previous backup.
- Identify failures after X amount of timeout/time passed and easy cleanup of file leftover (probably on T205627)
- Purge old metadata and make sure logs are rotated T205627
- Review and improve logging (beyond metadata)
- 1 retry after initial failure
- More optimization of certain database tables
- Maybe some kind of locking of backups and/or transfer.py to prevent concurrent actions on the same source or target servers
- Incremental/Differential backups T244884
- Document the last edit time (and potentially alert on) of some sample tables (e.g. recentchanges or revision) to verify the source databases are up to date (e.g. if its master, or intermediate master have replication stopped, or some other issue causing recent backups of stale data)
- Have a quick way to see which backup sources belong to each section (tendril, dashboard) (Done on puppet's site.pp
- Document and/or automate best server configuration for fast dump load (e.g. disable checksums, innodb transactionality, etc.)
- Enable the possibility of editing per-table options such as the engine and compression
- Workaround the "myloader doesn't import empty dbs" bug
- Binlog backups
- Easier/automated recovery/backup testing
Description
Details
Event Timeline
Change 579894 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly
Change 579894 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly
Change 579901 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly
Change 579901 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly
Change 582791 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies
Change 582791 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies
Change 583049 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Stop replication for s3 when snapshotting
Change 583049 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Stop replication for s3 when snapshotting
Change 584599 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] partman: Reference bacula recipe (unused to prevent accidental formatting)
Change 584599 merged by Jcrespo:
[operations/puppet@production] partman: Reference bacula recipe (unused to prevent accidental formatting)
Change 588668 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Increase max total size of Databases backups to 40 TB
Change 588668 merged by Jcrespo:
[operations/puppet@production] bacula: Increase max total size of Databases backups to 40 TB
Change 589266 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move all backup config templates to its own subdir
Change 589266 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move all backup config templates to its own subdir
Change 591309 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move OK alert description to wikitech
Change 591309 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move OK alert description to wikitech
Change 591326 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Check backups size also based on previous runs
Change 591326 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Check backups size also based on previous runs
Change 592599 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Tune backup size monitoring threasholds
Change 592599 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Tune backup size monitoring thresholds
Change 592608 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD
Change 592608 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD
Change 594096 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] transfer.py: Set timeout to 5 minutes
Change 594096 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] transfer.py: Set timeout to 5 minutes
Change 594099 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD
Change 594099 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD
Change 596189 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover zarcillo from db1115 to db2093
Change 596197 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dbtools: point zarcillo database scripts to use db2093
Change 596189 merged by Jcrespo:
[operations/puppet@production] mariadb: Switchover zarcillo from db1115 to db2093
Change 596197 merged by Jcrespo:
[operations/software@master] dbtools: point zarcillo database scripts to use db2093
Change 596242 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw
Change 596242 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw
Change 596251 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Allow zarcillo as a valid backup section
Change 596251 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Allow zarcillo as a valid backup section
Change 597005 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Disable monitoring screens on database backup hosts
Change 597005 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Disable monitoring screens on database backup hosts
Change 608053 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move transferpy deployment to debian package
Change 608053 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move transferpy deployment to debian package
Change 609101 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Fix remote_backup_mariadb.py deps and path
Change 609101 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Fix remote_backup_mariadb.py deps and path
Change 615155 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Adjust check parameters to get less false positives
Change 615155 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Adjust check parameters to get less false positives
Change 618720 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Reenable notifications for dbprov2003 after maintenance
Change 618722 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move x1 and misc logical dumps to dbprov1003
Change 618723 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] BackupStatistics: Do not raise an exception if metadata cannot be sent
Change 618723 abandoned by Jcrespo:
[operations/software/wmfmariadbpy@master] BackupStatistics: Do not raise an exception if metadata cannot be sent
Reason:
not a bug
Change 618720 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Reenable notifications for dbprov2003 after maintenance
Change 618722 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move x1 and misc logical dumps to dbprov1003
Change 620651 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Disable snapshots being sent to bacula
Change 620651 merged by Jcrespo:
[operations/puppet@production] mariadb: Disable snapshots being sent to bacula
Change 623525 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] mariadb-backups: Fix missing check on no ERROR msgs on logs
Change 623530 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs
Change 623532 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs
Change 623530 abandoned by Jcrespo:
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs
Reason:
duplicate
Change 623525 abandoned by Jcrespo:
[operations/software/wmfmariadbpy@master] mariadb-backups: Fix missing check on no ERROR msgs on logs
Reason:
duplicate
Change 623538 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update backup logic to check errors on log
Change 623538 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update backup logic to check errors on log
Change 623532 merged by jenkins-bot:
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs
Change 626172 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] remote_backup: Instead of using a preassigned port, autoselect one
Change 628163 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages
Change 628168 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] cli: Make /etc/wmfbackups the config dir for the main backup scripts
Change 628172 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] [WIP] Use the shared list of sections to validate backup checks
Change 626172 merged by Jcrespo:
[operations/software/wmfbackups@master] remote_backup: Instead of using a preassigned port, autoselect one
Change 628168 merged by Jcrespo:
[operations/software/wmfbackups@master] cli: Make /etc/wmfbackups the config dir for the main backup scripts
Change 629076 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Start using the package instead of the script on puppet
Change 629076 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Start using the package instead of the script on puppet
Change 629092 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Use the wmfbackups-remote package and not the puppet version
Change 628163 abandoned by Jcrespo:
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages
Reason:
split into 3 other patches, once per role
Change 629101 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages
Change 629101 merged by Jcrespo:
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages
Change 629092 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Use the wmfbackups-remote package and not the puppet version
Change 643220 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] Add new option, stats_file, to prevent sending passwords over the network/logs
Change 643223 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Make use of the new stats-file options, preventing password logging
Change 643300 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Make sure mydumper and xtrabackup are installed on the right hosts
Change 643220 merged by Jcrespo:
[operations/software/wmfbackups@master] Add new option, stats_file, to prevent sending passwords over the network/logs
Change 643223 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Use new stats-file option, preventing password logging
Change 643300 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Make sure mydumper and xtrabackup are installed
Change 662740 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] [WIP] Move database backups-related puppet code to its own profile/role
Change 663221 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[labs/private@master] dbbackups: Update password locations for database-backups db
Change 663221 merged by Jcrespo:
[labs/private@master] dbbackups: Update password locations for database-backups db
Change 662740 merged by Jcrespo:
[operations/puppet@production] dbbackups: Move database backups-related puppet code to its own profile/role
Change 663649 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbbackups: Create new puppet module dbbackups, move backup check to it
Change 663649 merged by Jcrespo:
[operations/puppet@production] dbbackups: Create new puppet module dbbackups, move backup check to it