Page MenuHomePhabricator
Create Task
Maniphest T138562

Improve regular production database backups handling
Open, MediumPublic
Actions

Description

  • Documentation, documentation, documentation
  • More flexibility: if possible per-table CSVs
  • More flexibility: Physical backups
  • Better recovery documentation: "one line to recover" (there is now a recover_section.py)
  • Faster point in time recovery/premade tools
  • Better compression
  • Prepare based on name of the backup, not just the section
  • Option optimization (e.g. double the use_memory)
  • More detailed health checks of backups (size, failures, objects, ...). E.g. check size is within a percentage of the previous backup.
  • Identify failures after X amount of timeout/time passed and easy cleanup of file leftover (probably on T205627)
  • Purge old metadata and make sure logs are rotated T205627
  • Review and improve logging (beyond metadata)
  • 1 retry after initial failure
  • More optimization of certain database tables
  • Maybe some kind of locking of backups and/or transfer.py to prevent concurrent actions on the same source or target servers
  • Incremental/Differential backups T244884
  • Document the last edit time (and potentially alert on) of some sample tables (e.g. recentchanges or revision) to verify the source databases are up to date (e.g. if its master, or intermediate master have replication stopped, or some other issue causing recent backups of stale data)
  • Have a quick way to see which backup sources belong to each section (tendril, dashboard) (Done on puppet's site.pp
  • Document and/or automate best server configuration for fast dump load (e.g. disable checksums, innodb transactionality, etc.)
  • Enable the possibility of editing per-table options such as the engine and compression
  • Workaround the "myloader doesn't import empty dbs" bug
  • Binlog backups
  • Easier/automated recovery/backup testing

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+56 -29dbbackups: Add enabled parameter backups on the cluster::management role
operations/software/wmfbackupsmaster+101 -0Add sql with the empty database structure to the repo
operations/puppetproduction+2 -2dbbackups: Reduce External store retention now that bacula is used long term
operations/puppetproduction+4 -4dbbackups: Move files and templates to dbbackups hierarchy
operations/puppetproduction+2 -0dbbackups: Limit concurrency of es backups on codfw, too
operations/puppetproduction+2 -2dbbackups: Create new puppet module dbbackups, move backup check to it
operations/puppetproduction+82 -82dbbackups: Move database backups-related puppet code to its own profile/role
labs/privatemaster+1 -2dbbackups: Update password locations for database-backups db
operations/puppetproduction+2 -2mariadb-backups: Make sure mydumper and xtrabackup are installed
operations/puppetproduction+24 -16mariadb-backups: Use new stats-file option, preventing password logging
operations/software/wmfbackupsmaster+53 -5Add new option, stats_file, to prevent sending passwords over the network/logs
operations/puppetproduction+7 -225mariadb-backups: Start using the package instead of the script on puppet
operations/puppetproduction+9 -462mariadb-backups: Use the wmfbackups-remote package and not the puppet version
operations/puppetproduction+8 -1 Kmariadb: Stop using puppet to deploy wmfbackups and use debian packages
operations/puppetproduction+15 -1 Kmariadb: Stop using puppet to deploy wmfbackups and use debian packages
operations/software/wmfbackupsmaster+4 -2cli: Make /etc/wmfbackups the config dir for the main backup scripts
operations/software/wmfbackupsmaster+5 -7remote_backup: Instead of using a preassigned port, autoselect one
operations/software/wmfbackupsmaster+1 -0[WIP] Use the shared list of sections to validate backup checks
operations/software/wmfbackupsmaster+10 -3mariadb-backups: Fix missing check on no ERROR msgs on logs
operations/puppetproduction+10 -3mariadb-backups: Update backup logic to check errors on log
operations/software/wmfmariadbpymaster+10 -3mariadb-backups: Fix missing check on no ERROR msgs on logs
operations/software/wmfbackupsmaster+10 -3mariadb-backups: Fix missing check on no ERROR msgs on logs
operations/puppetproduction+6 -3mariadb: Disable snapshots being sent to bacula
operations/puppetproduction+15 -19mariadb-backups: Move x1 and misc logical dumps to dbprov1003
operations/puppetproduction+0 -2mariadb-backups: Reenable notifications for dbprov2003 after maintenance
operations/software/wmfmariadbpymaster+2 -2BackupStatistics: Do not raise an exception if metadata cannot be sent
operations/puppetproduction+9 -9mariadb-backups: Adjust check parameters to get less false positives
operations/puppetproduction+3 -1mariadb-backups: Fix remote_backup_mariadb.py deps and path
operations/puppetproduction+2 -662mariadb-backups: Move transferpy deployment to debian package
operations/puppetproduction+2 -0mariadb-backups: Disable monitoring screens on database backup hosts
operations/puppetproduction+1 -1mariadb-backups: Allow zarcillo as a valid backup section
operations/puppetproduction+2 -2mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw
operations/softwaremaster+3 -3dbtools: point zarcillo database scripts to use db2093
operations/puppetproduction+29 -28mariadb: Switchover zarcillo from db1115 to db2093
operations/puppetproduction+3 -2mariadb-backups: Update transfer.py to HEAD
operations/software/wmfmariadbpymaster+2 -2transfer.py: Set timeout to 5 minutes
operations/puppetproduction+2 -2mariadb-backups: Update transfer.py to HEAD
operations/puppetproduction+19 -9mariadb-backups: Tune backup size monitoring thresholds
operations/puppetproduction+63 -22mariadb-backups: Check backups size also based on previous runs
operations/puppetproduction+8 -10mariadb-backups: Move OK alert description to wikitech
operations/puppetproduction+2 -2mariadb-backups: Move all backup config templates to its own subdir
operations/puppetproduction+1 -1bacula: Increase max total size of Databases backups to 40 TB
operations/puppetproduction+1 -0partman: Reference bacula recipe (unused to prevent accidental formatting)
operations/puppetproduction+3 -1mariadb-backups: Stop replication for s3 when snapshotting
operations/puppetproduction+1 -1mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies
operations/puppetproduction+1 -1mariadb-backups: Change bacula backup frequency of dbprov to weekly
operations/puppetproduction+1 -1mariadb-backups: Change bacula backup frequency of dbprov to weekly
operations/puppetproduction+10 -10mariadb-backups: Move x1 backups from dbprov[12]001 to dbprov[12]002
operations/puppetproduction+8 -2mariadb-backups: Increase snapshot frequency and retain those on bacula
operations/puppetproduction+0 -2mariadb-backups: Update code to wmfbackup HEAD to fix stalling issue
operations/puppetproduction+13 -12Correct invalid cron definition; add gtid to backups
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRobHT216175 HP Gen9 onboard controller review
 ResolvedfgiunchediT220787 Fix RAID handler alert and puppet facter to work with Gen10 hosts and ssacli tool
 OpenNoneT138562 Improve regular production database backups handling
 ResolvedjcrespoT131705 dbstore1001 low available space
 ResolvedjcrespoT111929 Puppetize grants for mysql hosts that are the source of recovery (dbstore, passive misc)
 ResolvedjcrespoT151999 Create script to monitor db dumps for backups are successful
 ResolvedjcrespoT79922 Set up backup strategy for es clusters
 DeclinedNoneT157668 Automate dataset backup recovery
 ResolvedjcrespoT162789 Create less overhead on bacula jobs when dumping production databases
 ResolvedjcrespoT169516 Implement cron-based mydumper backups on the dbstore role
 OpenNoneT151583 Use tls for dump backup generation
 OpenNoneT157702 Followup for TLS MariaDB server roll-out
 OpenNoneT152427 Create a check/calendar alert for MariaDB TLS certs
 ResolvedjcrespoT183469 Setup newer machines and replace all old misc (m*) and x1 eqiad machines
 ResolvedCmjohnsonT184054 Decommission db1029 and db1031
 ResolvedCmjohnsonT187542 Decommission db1043
 ResolvedCmjohnsonT184703 Decommission db1011
 ResolvedjcrespoT184704 Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw
Unknown Object (Task)
 ResolvedMarosteguiT186123 rack/setup/install db2093 (WAS: rack/setup/install tendril2001)
 ResolvedRobHT186172 switch port configuration for db2093
 ResolvedAndrewT189005 Failover m5 master from db1009 to db1073
 ResolvedCmjohnsonT189216 Decommission db1009
 ResolvedjcrespoT189655 Switchover m1 master from db1016 to db1063
 ResolvedCmjohnsonT190179 Decommission db1016
 ResolvedCmjohnsonT189773 Decommission db1020
 ResolvedjcrespoT189656 Switchover m2 master from db1020 to db1051
 ResolvedCmjohnsonT190262 Decommission db1001
 ResolvedjcrespoT183470 Setup newer machines and replace all old misc (m*) and x1 codfw machines
 ResolvedPapaulT187543 Decommission db2012
 ResolvedMarosteguiT187722 Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030)
 ResolvedRobHT187768 Decommission db2030
 ResolvedRobHT187886 Decommission db2011
 ResolvedjcrespoT169517 Research backup storage options and prepare a design document
 ResolvedjcrespoT184699 Generate consistent logical database backups in CODFW
 ResolvedNoneT183735 Check data consistency across production shards
 ResolvedMarosteguiT163190 Checksum data on s7
 ResolvedMarosteguiT165743 frwiktionary on s7 still needs fixing on the revision table
 ResolvedMarosteguiT162807 Run pt-table-checksum on s1 (enwiki)
 DeclinedMarosteguiT186503 Rebuild user_newtalk on db1052
 ResolvedmarkT184697 Failover existing eqiad database backup system to the new codfw database logical backup system
 ResolvedjcrespoT184696 Finish the database backups generation script to create consistent logical backups in CODFW
 ResolvedjcrespoT189383 Faster backups by archiving and unarchiving databases in parallel
 ResolvedjcrespoT189384 Faster backups by starting mydumper execution of different sections in parallel when they are about to finish and are running on a limited number of threads
 OpenNoneT156462 Framework to transfer files over the LAN
 ResolvedjcrespoT192358 Setup database logical backups on eqiad
 ResolvedjcrespoT198447 Monitor backup generation for failure or incorrect generation
 ResolvedjcrespoT198937 Setup database on tendril hosts to gather backup statistics
 ResolvedjcrespoT198987 Gather statistics about the backups on a database
 ResolvedjcrespoT203969 Create Icinga alerts on backup generation failure
 ResolvedjcrespoT201392 Finish eqiad metadata database backup setup (s1-s8, x1)
 DuplicateNoneT205626 Document clearly the mariadb backup and recovery setup
 OpenNoneT200398 Document clearly the mariadb backup and recovery setup, specially how to recover a backup
 OpenNoneT205627 Purge and monitor old metadata for the mariadb backups database
 OpenNoneT205628 Handle object metadata backups and compare it with stored database object inventory
 ResolvedLadsgroupT104459 Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas
 OpenNoneT57455 Provide dynamic report of differences between replica databases and production databases
 ResolvedMarosteguiT51199 Add index log_type_action
 ResolvedMarosteguiT217397 logging.log_title_time and logging.log_title_type_time indexes are not on tables.sql but they exist on most of the wikis
 ResolvedMarosteguiT250055 Remove image.img_deleted column from production
 ResolvedMarosteguiT250067 user_newtalk has two indexes not renamed in s4
 ResolvedMarosteguiT250062 ipb_parent_block_id_2 index on ipblocks table on s8 only
 ResolvedMarosteguiT250060 tl_namespace index on templatelinks is unique only in s8
 ResolvedReedyT249972 Wikitech has lots of database drifts with core and rest of the databases
 ResolvedMarosteguiT249973 db1110 has 5 important database drifts that are unique to the host
 ResolvedMarosteguiT250056 lc_lang_key index is lingering in production
 ResolvedMarosteguiT250057 type_acton index in logging table is lingering in production
 DeclinedNoneT250058 searchindex indexes are missing in production
 ResolvedMarosteguiT250063 inverse_timestamp column exists in text table, it shouldn't
 ResolvedMarosteguiT250066 text table still has old_* fields and indexes on some hosts
 ResolvedLadsgroupT250071 Rename ipb_address index on ipb_address to ipb_address_unique
 ResolvedMarosteguiT252126 tl_from index on templatelinks is lingering in production
 ResolvedMarosteguiT256679 text table in db1144 drifts from core considerably
 ResolvedMarosteguiT256680 imagelinks has index mismatch on s8
 ResolvedMarosteguiT256682 page_restrictions indexes have been majestically drifting from code
 ResolvedMarosteguiT256684 pl_from index still lingers in random hosts
 ResolvedMarosteguiT256685 pl_namespace index on pagelinks is unique only in s8
 ResolvedMarosteguiT260111 All sorts of random drifts in wikis in s3
 ResolvedLadsgroupT265321 ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production
 ResolvedMarosteguiT265349 querycache qc_type and qc_title have different nullabality on s1 only
 ResolvedMarosteguiT265445 Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5
 ResolvedLadsgroupT269348 wikitech database has almost all of its varbinary fields wrong
 StalledKormatT277118 iw_url in interwiki is varbinary(127) in production but blob in code
 OpenNoneT277354 "chemical" major mime type was never added to production database
 StalledKormatT278619 ipb_timestamp is varbinary(14) in old wikis while being binary(14) in the code since 2007
 ResolvedMarosteguiT278621 job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007
 StalledKormatT277116 fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production
 ResolvedjcrespoT206203 Implement database binary backups into the production infrastructure
 ResolvedjcrespoT206204 Research options for producing binary backups (lvm snapshots, cold backups, mariabackup)
 ResolvedjcrespoT210292 Implement a proof of concept of a snapshot cycle automation for a mediawiki section database
 ResolvedjcrespoT213406 Purchase and setup remaining hosts for database backups
 ResolvedjcrespoT213404 Design the final architecture for the database binary backups
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
 ResolvedjcrespoT220572 Productionize eqiad and codfw source backup hosts & codfw backup test host
 ResolvedPapaulT219461 rack/setup/install db2102.codfw.wmnet as a testing host for codfw backups
 DeclinedNoneT218188 Import issue (bug?) on Python 3.4/3.5 + multiprocessing affecting Cumin
 ResolvedGuozr.imT218189 wmfmariadbpy/CuminExecution must capture Exception cumin.transports.WorkerError
 ResolvedjcrespoT219631 Create a recovery/provisioning script for database binary backups
 ResolvedPapaulT220002 Decommission dbstore2001.codfw.wmnet and dbstore2002.codfw.wmnet
 ResolvedjcrespoT219399 rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts
 ResolvedPapaulT219463 rack/setup/install (5) codfw dedicated dump slaves
 ResolvedCmjohnsonT218985 rack/setup/install db1139|db1140.eqiad.wmnet (2 dump slaves)
Unknown Object (Task)
 ResolvedPapaulT218336 rack/setup/deploy codfw dedicated backup recovery/provisioning hosts
Unknown Object (Task)
 ResolvedjcrespoT225378 db2097 (codfw s1&s6 source backups) mariadb@s6 *process* (10.1.39) crashed on 2019-06-08
 ResolvedjcrespoT236924 dbprov2002 slower to generate snapshots
 ResolvedL0st3xpl0r3rT248661 Fix return code handling on transfer.py error
 ResolvedjcrespoT257551 Expand database provisioning/backup service to accomodate for growing capacity and high availability needs
Unknown Object (Task)
 ResolvedPapaulT258749 (Need By: 2020-09-14) rack/setup/install dbprov2003.codfw.wmnet
Unknown Object (Task)
 ResolvedCmjohnsonT258750 (2020-08-15) rack/setup/install dbprov1003.eqiad.wmnet
Unknown Object (Task)
 ResolvedPapaulT260819 (Need By: 2020-09-15) rack/setup/install db2141 (or next in sequence)
Unknown Object (Task)
 ResolvedCmjohnsonT260817 (Need By: 2020-09-15) rack/setup/install db1150 (see note on hostname)
 OpenNoneT244884 Implement logic to be able to perform incremental backups of ES hosts
 ResolvedjcrespoT282249 Setup backup1003 and backup2003 as the storage location for es bacula backups

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
L0st3xpl0r3r mentioned this in T248590: [GSoC 2020 Proposal] Making a supervisor for monitoring the status of all the Wikimedia database instances.Apr 2 2020, 6:08 AM
L0st3xpl0r3r added a subscriber: L0st3xpl0r3r.Apr 2 2020, 6:29 AM
jcrespo mentioned this in T159524: backup space is used unwisely.Apr 3 2020, 10:25 AM
gerritbot added a comment.Apr 14 2020, 10:57 AM

Change 588668 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Increase max total size of Databases backups to 40 TB

https://gerrit.wikimedia.org/r/588668

gerritbot added a comment.Apr 15 2020, 8:48 AM

Change 588668 merged by Jcrespo:
[operations/puppet@production] bacula: Increase max total size of Databases backups to 40 TB

https://gerrit.wikimedia.org/r/588668

gerritbot added a comment.Apr 16 2020, 8:15 AM

Change 589266 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move all backup config templates to its own subdir

https://gerrit.wikimedia.org/r/589266

gerritbot added a comment.Apr 16 2020, 8:54 AM

Change 589266 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move all backup config templates to its own subdir

https://gerrit.wikimedia.org/r/589266

gerritbot added a comment.Apr 21 2020, 9:24 AM

Change 591309 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move OK alert description to wikitech

https://gerrit.wikimedia.org/r/591309

gerritbot added a comment.Apr 21 2020, 10:18 AM

Change 591309 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move OK alert description to wikitech

https://gerrit.wikimedia.org/r/591309

gerritbot added a comment.Apr 21 2020, 12:30 PM

Change 591326 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Check backups size also based on previous runs

https://gerrit.wikimedia.org/r/591326

gerritbot added a comment.Apr 27 2020, 6:55 AM

Change 591326 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Check backups size also based on previous runs

https://gerrit.wikimedia.org/r/591326

gerritbot added a comment.Apr 27 2020, 7:22 AM

Change 592599 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Tune backup size monitoring threasholds

https://gerrit.wikimedia.org/r/592599

gerritbot added a comment.Apr 27 2020, 8:11 AM

Change 592599 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Tune backup size monitoring thresholds

https://gerrit.wikimedia.org/r/592599

jcrespo updated the task description. (Show Details)Apr 27 2020, 8:20 AM
gerritbot added a comment.Apr 27 2020, 8:35 AM

Change 592608 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/592608

gerritbot added a comment.Apr 27 2020, 8:40 AM

Change 592608 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/592608

gerritbot added a comment.May 4 2020, 8:03 AM

Change 594096 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] transfer.py: Set timeout to 5 minutes

https://gerrit.wikimedia.org/r/594096

gerritbot added a comment.May 4 2020, 8:13 AM

Change 594096 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] transfer.py: Set timeout to 5 minutes

https://gerrit.wikimedia.org/r/594096

gerritbot added a comment.May 4 2020, 8:21 AM

Change 594099 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/594099

gerritbot added a comment.May 4 2020, 8:40 AM

Change 594099 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/594099

gerritbot added a comment.May 13 2020, 11:51 AM

Change 596189 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover zarcillo from db1115 to db2093

https://gerrit.wikimedia.org/r/596189

gerritbot added a comment.May 13 2020, 12:18 PM

Change 596197 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dbtools: point zarcillo database scripts to use db2093

https://gerrit.wikimedia.org/r/596197

gerritbot added a comment.May 13 2020, 2:57 PM

Change 596189 merged by Jcrespo:
[operations/puppet@production] mariadb: Switchover zarcillo from db1115 to db2093

https://gerrit.wikimedia.org/r/596189

gerritbot added a comment.May 13 2020, 3:54 PM

Change 596197 merged by Jcrespo:
[operations/software@master] dbtools: point zarcillo database scripts to use db2093

https://gerrit.wikimedia.org/r/596197

gerritbot added a comment.May 13 2020, 3:59 PM

Change 596242 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw

https://gerrit.wikimedia.org/r/596242

gerritbot added a comment.May 13 2020, 4:02 PM

Change 596242 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw

https://gerrit.wikimedia.org/r/596242

gerritbot added a comment.May 13 2020, 5:23 PM

Change 596251 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Allow zarcillo as a valid backup section

https://gerrit.wikimedia.org/r/596251

gerritbot added a comment.May 13 2020, 5:24 PM

Change 596251 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Allow zarcillo as a valid backup section

https://gerrit.wikimedia.org/r/596251

gerritbot added a comment.May 18 2020, 8:15 AM

Change 597005 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Disable monitoring screens on database backup hosts

https://gerrit.wikimedia.org/r/597005

gerritbot added a comment.May 18 2020, 8:15 AM

Change 597005 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Disable monitoring screens on database backup hosts

https://gerrit.wikimedia.org/r/597005

gerritbot added a comment.Jun 26 2020, 2:13 PM

Change 608053 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move transferpy deployment to debian package

https://gerrit.wikimedia.org/r/608053

gerritbot added a comment.Jun 29 2020, 8:19 AM

Change 608053 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move transferpy deployment to debian package

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608053

gerritbot added a comment.Jul 2 2020, 7:45 AM

Change 609101 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Fix remote_backup_mariadb.py deps and path

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609101

gerritbot added a comment.Jul 2 2020, 7:48 AM

Change 609101 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Fix remote_backup_mariadb.py deps and path

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609101

gerritbot added a comment.Jul 21 2020, 9:02 AM

Change 615155 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Adjust check parameters to get less false positives

https://gerrit.wikimedia.org/r/615155

gerritbot added a comment.Jul 21 2020, 9:18 AM

Change 615155 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Adjust check parameters to get less false positives

https://gerrit.wikimedia.org/r/615155

gerritbot added a comment.Aug 6 2020, 9:20 AM

Change 618720 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Reenable notifications for dbprov2003 after maintenance

https://gerrit.wikimedia.org/r/618720

gerritbot added a comment.Aug 6 2020, 9:40 AM

Change 618722 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move x1 and misc logical dumps to dbprov1003

https://gerrit.wikimedia.org/r/618722

gerritbot added a comment.Aug 6 2020, 9:47 AM

Change 618723 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] BackupStatistics: Do not raise an exception if metadata cannot be sent

https://gerrit.wikimedia.org/r/618723

gerritbot added a comment.Aug 6 2020, 9:53 AM

Change 618723 abandoned by Jcrespo:
[operations/software/wmfmariadbpy@master] BackupStatistics: Do not raise an exception if metadata cannot be sent

Reason:
not a bug

https://gerrit.wikimedia.org/r/618723

gerritbot added a comment.Aug 6 2020, 10:47 AM

Change 618720 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Reenable notifications for dbprov2003 after maintenance

https://gerrit.wikimedia.org/r/618720

gerritbot added a comment.Aug 6 2020, 1:49 PM

Change 618722 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move x1 and misc logical dumps to dbprov1003

https://gerrit.wikimedia.org/r/618722

gerritbot added a comment.Aug 17 2020, 8:13 AM

Change 620651 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Disable snapshots being sent to bacula

https://gerrit.wikimedia.org/r/620651

gerritbot added a comment.Aug 17 2020, 8:20 AM

Change 620651 merged by Jcrespo:
[operations/puppet@production] mariadb: Disable snapshots being sent to bacula

https://gerrit.wikimedia.org/r/620651

gerritbot added a comment.Sep 1 2020, 8:01 AM

Change 623525 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] mariadb-backups: Fix missing check on no ERROR msgs on logs

https://gerrit.wikimedia.org/r/623525

jcrespo mentioned this in rOSWB777d5f023a67: transfer.py: Set timeout to 5 minutes.Sep 1 2020, 8:36 AM
gerritbot added a comment.Sep 1 2020, 8:38 AM

Change 623530 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs

https://gerrit.wikimedia.org/r/623530

gerritbot added a comment.Sep 1 2020, 8:39 AM

Change 623532 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs

https://gerrit.wikimedia.org/r/623532

gerritbot added a comment.Sep 1 2020, 8:40 AM

Change 623530 abandoned by Jcrespo:
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs

Reason:
duplicate

https://gerrit.wikimedia.org/r/623530

gerritbot added a comment.Sep 1 2020, 8:40 AM

Change 623525 abandoned by Jcrespo:
[operations/software/wmfmariadbpy@master] mariadb-backups: Fix missing check on no ERROR msgs on logs

Reason:
duplicate

https://gerrit.wikimedia.org/r/623525

gerritbot added a comment.Sep 1 2020, 8:51 AM

Change 623538 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update backup logic to check errors on log

https://gerrit.wikimedia.org/r/623538

gerritbot added a comment.Sep 1 2020, 8:54 AM

Change 623538 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update backup logic to check errors on log

https://gerrit.wikimedia.org/r/623538

gerritbot added a comment.Sep 3 2020, 7:45 AM

Change 623532 merged by jenkins-bot:
[operations/software/wmfbackups@master] mariadb-backups: Fix missing check on no ERROR msgs on logs

https://gerrit.wikimedia.org/r/623532

gerritbot added a comment.Sep 9 2020, 3:12 PM

Change 626172 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] remote_backup: Instead of using a preassigned port, autoselect one

https://gerrit.wikimedia.org/r/626172

jcrespo updated the task description. (Show Details)Sep 9 2020, 3:13 PM
gerritbot added a comment.Sep 17 2020, 6:08 PM

Change 628163 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages

https://gerrit.wikimedia.org/r/628163

gerritbot added a comment.Sep 17 2020, 6:22 PM

Change 628168 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] cli: Make /etc/wmfbackups the config dir for the main backup scripts

https://gerrit.wikimedia.org/r/628168

gerritbot added a comment.Sep 17 2020, 6:30 PM

Change 628172 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] [WIP] Use the shared list of sections to validate backup checks

https://gerrit.wikimedia.org/r/628172

gerritbot added a comment.Sep 18 2020, 1:07 PM

Change 626172 merged by Jcrespo:
[operations/software/wmfbackups@master] remote_backup: Instead of using a preassigned port, autoselect one

https://gerrit.wikimedia.org/r/626172

gerritbot added a comment.Sep 18 2020, 1:08 PM

Change 628168 merged by Jcrespo:
[operations/software/wmfbackups@master] cli: Make /etc/wmfbackups the config dir for the main backup scripts

https://gerrit.wikimedia.org/r/628168

gerritbot added a comment.Sep 22 2020, 10:04 AM

Change 629076 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Start using the package instead of the script on puppet

https://gerrit.wikimedia.org/r/629076

gerritbot added a comment.Sep 22 2020, 10:43 AM

Change 629076 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Start using the package instead of the script on puppet

https://gerrit.wikimedia.org/r/629076

gerritbot added a comment.Sep 22 2020, 11:39 AM

Change 629092 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Use the wmfbackups-remote package and not the puppet version

https://gerrit.wikimedia.org/r/629092

gerritbot added a comment.Sep 22 2020, 12:03 PM

Change 628163 abandoned by Jcrespo:
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages

Reason:
split into 3 other patches, once per role

https://gerrit.wikimedia.org/r/628163

gerritbot added a comment.Sep 22 2020, 12:04 PM

Change 629101 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages

https://gerrit.wikimedia.org/r/629101

gerritbot added a comment.Sep 22 2020, 12:33 PM

Change 629101 merged by Jcrespo:
[operations/puppet@production] mariadb: Stop using puppet to deploy wmfbackups and use debian packages

https://gerrit.wikimedia.org/r/629101

gerritbot added a comment.Sep 22 2020, 1:13 PM

Change 629092 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Use the wmfbackups-remote package and not the puppet version

https://gerrit.wikimedia.org/r/629092

jcrespo closed subtask T257551: Expand database provisioning/backup service to accomodate for growing capacity and high availability needs as Resolved.Sep 29 2020, 8:44 AM
jcrespo updated the task description. (Show Details)Sep 29 2020, 10:58 AM
LSobanski edited projects, added Data-Persistence-Backup; removed DBA.Oct 6 2020, 8:25 AM
gerritbot added a comment.Nov 24 2020, 10:10 AM

Change 643220 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] Add new option, stats_file, to prevent sending passwords over the network/logs

https://gerrit.wikimedia.org/r/643220

gerritbot added a comment.Nov 24 2020, 10:24 AM

Change 643223 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Make use of the new stats-file options, preventing password logging

https://gerrit.wikimedia.org/r/643223

gerritbot added a comment.Nov 24 2020, 4:58 PM

Change 643300 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Make sure mydumper and xtrabackup are installed on the right hosts

https://gerrit.wikimedia.org/r/643300

gerritbot added a comment.Nov 25 2020, 10:07 AM

Change 643220 merged by Jcrespo:
[operations/software/wmfbackups@master] Add new option, stats_file, to prevent sending passwords over the network/logs

https://gerrit.wikimedia.org/r/643220

gerritbot added a comment.Nov 25 2020, 4:25 PM

Change 643223 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Use new stats-file option, preventing password logging

https://gerrit.wikimedia.org/r/643223

gerritbot added a comment.Nov 25 2020, 4:27 PM

Change 643300 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Make sure mydumper and xtrabackup are installed

https://gerrit.wikimedia.org/r/643300

jcrespo added a subtask: T244884: Implement logic to be able to perform incremental backups of ES hosts.Feb 8 2021, 1:30 PM
jcrespo updated the task description. (Show Details)Feb 8 2021, 1:33 PM
jcrespo added a subscriber: LSobanski.
gerritbot added a comment.Feb 8 2021, 4:44 PM

Change 662740 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] [WIP] Move database backups-related puppet code to its own profile/role

https://gerrit.wikimedia.org/r/662740

gerritbot added a comment.Feb 10 2021, 3:29 PM

Change 663221 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[labs/private@master] dbbackups: Update password locations for database-backups db

https://gerrit.wikimedia.org/r/663221

gerritbot added a comment.Feb 10 2021, 3:44 PM

Change 663221 merged by Jcrespo:
[labs/private@master] dbbackups: Update password locations for database-backups db

https://gerrit.wikimedia.org/r/663221

jcrespo mentioned this in rLPRI4d446703f54c: dbbackups: Update password locations for database-backups db.Feb 10 2021, 3:45 PM
gerritbot added a comment.Feb 11 2021, 6:09 PM

Change 662740 merged by Jcrespo:
[operations/puppet@production] dbbackups: Move database backups-related puppet code to its own profile/role

https://gerrit.wikimedia.org/r/662740

gerritbot added a comment.Feb 11 2021, 6:24 PM

Change 663649 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbbackups: Create new puppet module dbbackups, move backup check to it

https://gerrit.wikimedia.org/r/663649

gerritbot added a comment.Feb 11 2021, 6:42 PM

Change 663649 merged by Jcrespo:
[operations/puppet@production] dbbackups: Create new puppet module dbbackups, move backup check to it

https://gerrit.wikimedia.org/r/663649

jcrespo closed subtask T79922: Set up backup strategy for es clusters as Resolved.Feb 16 2021, 3:08 PM
jcrespo closed subtask T111929: Puppetize grants for mysql hosts that are the source of recovery (dbstore, passive misc) as Resolved.Mar 2 2021, 3:19 PM
gerritbot added a comment.Mar 9 2021, 1:57 PM

Change 670174 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbbackups: Limit concurrency of es backups on codfw, too

https://gerrit.wikimedia.org/r/670174

gerritbot added a comment.Mar 9 2021, 3:52 PM

Change 670217 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbbackups: Move files and templates to dbbackups hierarchy

https://gerrit.wikimedia.org/r/670217

gerritbot added a comment.Mar 10 2021, 9:52 AM

Change 670174 merged by Jcrespo:
[operations/puppet@production] dbbackups: Limit concurrency of es backups on codfw, too

https://gerrit.wikimedia.org/r/670174

gerritbot added a comment.Mar 10 2021, 10:03 AM

Change 670217 merged by Jcrespo:
[operations/puppet@production] dbbackups: Move files and templates to dbbackups hierarchy

https://gerrit.wikimedia.org/r/670217

gerritbot added a comment.Mar 12 2021, 10:45 AM

Change 671112 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbbackups: Reduce External store retention now that bacula store it long term

https://gerrit.wikimedia.org/r/671112

gerritbot added a comment.Mar 12 2021, 11:04 AM

Change 671112 merged by Jcrespo:
[operations/puppet@production] dbbackups: Reduce External store retention now that bacula is used long term

https://gerrit.wikimedia.org/r/671112

gerritbot added a comment.Mar 18 2021, 3:21 PM

Change 673292 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfbackups@master] Add sql with the empty database structure to the repo

https://gerrit.wikimedia.org/r/673292

gerritbot added a comment.Mar 19 2021, 9:33 AM

Change 673292 merged by jenkins-bot:
[operations/software/wmfbackups@master] Add sql with the empty database structure to the repo

https://gerrit.wikimedia.org/r/673292

gerritbot added a comment.Apr 20 2021, 9:37 AM

Change 681317 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Add enabled parameter backups on the cluster::management role

https://gerrit.wikimedia.org/r/681317

gerritbot added a comment.Apr 20 2021, 10:33 AM

Change 681317 merged by Jcrespo:

[operations/puppet@production] dbbackups: Add enabled parameter backups on the cluster::management role

https://gerrit.wikimedia.org/r/681317

jcrespo closed subtask T282249: Setup backup1003 and backup2003 as the storage location for es bacula backups as Resolved.May 29 2021, 10:07 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL