Page MenuHomePhabricator
Create Task
Maniphest T138562

Improve regular production database backups handling
Open, MediumPublic
Actions

Description

  • Documentation, documentation, documentation
  • More flexibility: if possible per-table CSVs
  • More flexibility: Physical backups
  • Better recovery documentation: "one line to recover" (there is now a recover_section.py)
  • Faster point in time recovery/premade tools
  • Better compression
  • Prepare based on name of the backup, not just the section
  • Option optimization (e.g. double the use_memory)
  • More detailed health checks of backups (size, failures, objects, ...). E.g. check size is within a percentage of the previous backup.
  • Identify failures after X amount of timeout/time passed and easy cleanup of file leftover (probably on T205627)
  • Purge old metadata and make sure logs are rotated T205627
  • Review and improve logging (beyond metadata)
  • 1 retry after initial failure
  • More optimization of certain database tables
  • Maybe some kind of locking of backups and/or transfer.py to prevent concurrent actions on the same source or target servers
  • Differential backups
  • Document the last edit time (and potentially alert on) of some sample tables (e.g. recentchanges or revision) to verify the source databases are up to date (e.g. if its master, or intermediate master have replication stopped, or some other issue causing recent backups of stale data)
  • Have a quick way to see which backup sources belong to each section (tendril, dashboard)
  • Document and/or automate best server configuration for fast dump load (e.g. disable checksums, innodb transactionality, etc.)
  • Enable the possibility of editing per-table options such as the engine and compression
  • Workaround the "myloader doesn't import empty dbs" bug

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+15 -19mariadb-backups: Move x1 and misc logical dumps to dbprov1003
operations/puppetproduction+0 -2mariadb-backups: Reenable notifications for dbprov2003 after maintenance
operations/software/wmfmariadbpymaster+2 -2BackupStatistics: Do not raise an exception if metadata cannot be sent
operations/puppetproduction+9 -9mariadb-backups: Adjust check parameters to get less false positives
operations/puppetproduction+3 -1mariadb-backups: Fix remote_backup_mariadb.py deps and path
operations/puppetproduction+2 -662mariadb-backups: Move transferpy deployment to debian package
operations/puppetproduction+2 -0mariadb-backups: Disable monitoring screens on database backup hosts
operations/puppetproduction+1 -1mariadb-backups: Allow zarcillo as a valid backup section
operations/puppetproduction+2 -2mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw
operations/softwaremaster+3 -3dbtools: point zarcillo database scripts to use db2093
operations/puppetproduction+29 -28mariadb: Switchover zarcillo from db1115 to db2093
operations/puppetproduction+3 -2mariadb-backups: Update transfer.py to HEAD
operations/software/wmfmariadbpymaster+2 -2transfer.py: Set timeout to 5 minutes
operations/puppetproduction+2 -2mariadb-backups: Update transfer.py to HEAD
operations/puppetproduction+19 -9mariadb-backups: Tune backup size monitoring thresholds
operations/puppetproduction+63 -22mariadb-backups: Check backups size also based on previous runs
operations/puppetproduction+8 -10mariadb-backups: Move OK alert description to wikitech
operations/puppetproduction+2 -2mariadb-backups: Move all backup config templates to its own subdir
operations/puppetproduction+1 -1bacula: Increase max total size of Databases backups to 40 TB
operations/puppetproduction+1 -0partman: Reference bacula recipe (unused to prevent accidental formatting)
operations/puppetproduction+3 -1mariadb-backups: Stop replication for s3 when snapshotting
operations/puppetproduction+1 -1mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies
operations/puppetproduction+1 -1mariadb-backups: Change bacula backup frequency of dbprov to weekly
operations/puppetproduction+1 -1mariadb-backups: Change bacula backup frequency of dbprov to weekly
operations/puppetproduction+10 -10mariadb-backups: Move x1 backups from dbprov[12]001 to dbprov[12]002
operations/puppetproduction+8 -2mariadb-backups: Increase snapshot frequency and retain those on bacula
operations/puppetproduction+0 -2mariadb-backups: Update code to wmfbackup HEAD to fix stalling issue
operations/puppetproduction+13 -12Correct invalid cron definition; add gtid to backups
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRobHT216175 HP Gen9 onboard controller review
 ResolvedfgiunchediT220787 Fix RAID handler alert and puppet facter to work with Gen10 hosts and ssacli tool
 OpenNoneT138562 Improve regular production database backups handling
 ResolvedjcrespoT131705 dbstore1001 low available space
 OpenNoneT111929 Puppetize grants for mysql hosts that are the source of recovery (dbstore, passive misc)
 ResolvedjcrespoT151999 Create script to monitor db dumps for backups are successful
 OpenjcrespoT79922 Set up backup strategy for es clusters
 StalledNoneT244884 Implement logic to be able to perform full and incremental backups of ES hosts
 DeclinedNoneT157668 Automate dataset backup recovery
 ResolvedjcrespoT162789 Create less overhead on bacula jobs when dumping production databases
 ResolvedjcrespoT169516 Implement cron-based mydumper backups on the dbstore role
 OpenNoneT151583 Use tls for dump backup generation
 OpenNoneT157702 Followup for TLS MariaDB server roll-out
 OpenNoneT152427 Create a check/calendar alert for MariaDB TLS certs
 ResolvedjcrespoT183469 Setup newer machines and replace all old misc (m*) and x1 eqiad machines
 ResolvedCmjohnsonT184054 Decommission db1029 and db1031
 ResolvedCmjohnsonT187542 Decommission db1043
 ResolvedCmjohnsonT184703 Decommission db1011
 ResolvedjcrespoT184704 Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw
Unknown Object (Task)
 Resolved• MarosteguiT186123 rack/setup/install db2093 (WAS: rack/setup/install tendril2001)
 ResolvedRobHT186172 switch port configuration for db2093
 ResolvedAndrewT189005 Failover m5 master from db1009 to db1073
 ResolvedCmjohnsonT189216 Decommission db1009
 ResolvedjcrespoT189655 Switchover m1 master from db1016 to db1063
 ResolvedCmjohnsonT190179 Decommission db1016
 ResolvedCmjohnsonT189773 Decommission db1020
 ResolvedjcrespoT189656 Switchover m2 master from db1020 to db1051
 ResolvedCmjohnsonT190262 Decommission db1001
 ResolvedjcrespoT183470 Setup newer machines and replace all old misc (m*) and x1 codfw machines
 ResolvedPapaulT187543 Decommission db2012
 Resolved• MarosteguiT187722 Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030)
 ResolvedRobHT187768 Decommission db2030
 ResolvedRobHT187886 Decommission db2011
 ResolvedjcrespoT169517 Research backup storage options and prepare a design document
 ResolvedjcrespoT184699 Generate consistent logical database backups in CODFW
 ResolvedNoneT183735 Check data consistency across production shards
 Resolved• MarosteguiT163190 Checksum data on s7
 Resolved• MarosteguiT165743 frwiktionary on s7 still needs fixing on the revision table
 Resolved• MarosteguiT162807 Run pt-table-checksum on s1 (enwiki)
 Declined• MarosteguiT186503 Rebuild user_newtalk on db1052
 ResolvedmarkT184697 Failover existing eqiad database backup system to the new codfw database logical backup system
 ResolvedjcrespoT184696 Finish the database backups generation script to create consistent logical backups in CODFW
 ResolvedjcrespoT189383 Faster backups by archiving and unarchiving databases in parallel
 ResolvedjcrespoT189384 Faster backups by starting mydumper execution of different sections in parallel when they are about to finish and are running on a limited number of threads
 OpenNoneT156462 Framework to transfer files over the LAN
 ResolvedjcrespoT192358 Setup database logical backups on eqiad
 ResolvedjcrespoT198447 Monitor backup generation for failure or incorrect generation
 ResolvedjcrespoT198937 Setup database on tendril hosts to gather backup statistics
 ResolvedjcrespoT198987 Gather statistics about the backups on a database
 ResolvedjcrespoT203969 Create Icinga alerts on backup generation failure
 ResolvedjcrespoT201392 Finish eqiad metadata database backup setup (s1-s8, x1)
 DuplicateNoneT205626 Document clearly the mariadb backup and recovery setup
 OpenNoneT200398 Document clearly the mariadb backup and recovery setup, specially how to recover a backup
 OpenNoneT205627 Purge and monitor old metadata for the mariadb backups database
 OpenNoneT205628 Handle object metadata backups and compare it with stored database object inventory
 OpenNoneT104459 Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves
 OpenNoneT57455 Provide dynamic report of differences between replica databases and production databases
 Resolved• MarosteguiT51199 Add index log_type_action
 Resolved• MarosteguiT217397 logging.log_title_time and logging.log_title_type_time indexes are not on tables.sql but they exist on most of the wikis
 Resolved• MarosteguiT250055 Remove image.img_deleted column from production
 Resolved• MarosteguiT250067 user_newtalk has two indexes not renamed in s4
 Resolved• MarosteguiT250062 ipb_parent_block_id_2 index on ipblocks table on s8 only
 Resolved• MarosteguiT250060 tl_namespace index on templatelinks is unique only in s8
 ResolvedReedyT249972 Wikitech has lots of database drifts with core and rest of the databases
 Resolved• MarosteguiT249973 db1110 has 5 important database drifts that are unique to the host
 Resolved• MarosteguiT250056 lc_lang_key index is lingering in production
 Resolved• MarosteguiT250057 type_acton index in logging table is lingering in production
 DeclinedNoneT250058 searchindex indexes are missing in production
 Resolved• MarosteguiT250063 inverse_timestamp column exists in text table, it shouldn't
 Resolved• MarosteguiT250066 text table still has old_* fields and indexes on some hosts
 ResolvedLadsgroupT250071 Rename ipb_address index on ipb_address to ipb_address_unique
 Resolved• MarosteguiT252126 tl_from index on templatelinks is lingering in production
 Resolved• MarosteguiT256679 text table in db1144 drifts from core considerably
 Resolved• MarosteguiT256680 imagelinks has index mismatch on s8
 Resolved• MarosteguiT256682 page_restrictions indexes have been majestically drifting from code
 Resolved• MarosteguiT256684 pl_from index still lingers in random hosts
 Stalled• MarosteguiT256685 pl_namespace index on pagelinks is unique only in s8
 OpenNoneT260111 All sorts of random drifts in wikis in s3
 ResolvedjcrespoT206203 Implement database binary backups into the production infrastructure
 ResolvedjcrespoT206204 Research options for producing binary backups (lvm snapshots, cold backups, mariabackup)
 ResolvedjcrespoT210292 Implement a proof of concept of a snapshot cycle automation for a mediawiki section database
 ResolvedjcrespoT213406 Purchase and setup remaining hosts for database backups
 ResolvedjcrespoT213404 Design the final architecture for the database binary backups
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
 ResolvedjcrespoT220572 Productionize eqiad and codfw source backup hosts & codfw backup test host
 ResolvedPapaulT219461 rack/setup/install db2102.codfw.wmnet as a testing host for codfw backups
 DeclinedNoneT218188 Import issue (bug?) on Python 3.4/3.5 + multiprocessing affecting Cumin
 ResolvedGuozr.imT218189 wmfmariadbpy/CuminExecution must capture Exception cumin.transports.WorkerError
 ResolvedjcrespoT219631 Create a recovery/provisioning script for database binary backups
 ResolvedPapaulT220002 Decommission dbstore2001.codfw.wmnet and dbstore2002.codfw.wmnet
 ResolvedjcrespoT219399 rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts
 ResolvedPapaulT219463 rack/setup/install (5) codfw dedicated dump slaves
 ResolvedCmjohnsonT218985 rack/setup/install db1139|db1140.eqiad.wmnet (2 dump slaves)
Unknown Object (Task)
 ResolvedPapaulT218336 rack/setup/deploy codfw dedicated backup recovery/provisioning hosts
Unknown Object (Task)
 ResolvedjcrespoT225378 db2097 (codfw s1&s6 source backups) mariadb@s6 *process* (10.1.39) crashed on 2019-06-08
 ResolvedjcrespoT236924 dbprov2002 slower to generate snapshots
 ResolvedL0st3xpl0r3rT248661 Fix return code handling on transfer.py error
 OpenjcrespoT257551 Expand database provisioning/backup service to accomodate for growing capacity and high availability needs
Unknown Object (Task)
 ResolvedPapaulT258749 (Need By: 2020-09-14) rack/setup/install dbprov2003.codfw.wmnet
Unknown Object (Task)
 OpenCmjohnsonT258750 (2020-08-15) rack/setup/install dbprov1003.eqiad.wmnet
Unknown Object (Task)
Unknown Object (Task)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jcrespo renamed this task from Improve db backup handling to Improve regular production database backups handling.Apr 12 2017, 10:26 AM
jcrespo raised the priority of this task from Low to Medium.
jcrespo added subtasks: T111929: Puppetize grants for mysql hosts that are the source of recovery (dbstore, passive misc), T151999: Create script to monitor db dumps for backups are successful, T79922: Set up backup strategy for es clusters, T157668: Automate dataset backup recovery.
jcrespo moved this task from Backlog to Meta/Epic on the DBA board.
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)Mar 9 2018, 3:16 PM
greg removed a subscriber: greg.Apr 11 2019, 9:20 PM
jcrespo updated the task description. (Show Details)Jun 25 2019, 8:58 AM
jcrespo updated the task description. (Show Details)Dec 2 2019, 10:03 AM
jcrespo updated the task description. (Show Details)Dec 10 2019, 3:13 PM
jcrespo updated the task description. (Show Details)
gerritbot added a comment.Feb 21 2020, 9:34 AM

Change 573961 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update code to wmfbackup HEAD to fix stalling issue

https://gerrit.wikimedia.org/r/573961

gerritbot added a comment.Feb 21 2020, 9:41 AM

Change 573961 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update code to wmfbackup HEAD to fix stalling issue

https://gerrit.wikimedia.org/r/573961

gerritbot added a comment.Mar 6 2020, 7:52 AM

Change 577462 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Increase snapshot frequency and retention (6 days)

https://gerrit.wikimedia.org/r/577462

gerritbot added a comment.Mar 10 2020, 8:22 AM

Change 577462 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Increase snapshot frequency and retain those on bacula

https://gerrit.wikimedia.org/r/577462

gerritbot added a comment.Mar 16 2020, 8:39 AM

Change 579892 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move x1 backups from dbprov[12]001 to dbprov[12]002

https://gerrit.wikimedia.org/r/579892

gerritbot added a comment.Mar 16 2020, 8:59 AM

Change 579892 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move x1 backups from dbprov[12]001 to dbprov[12]002

https://gerrit.wikimedia.org/r/579892

gerritbot added a comment.Mar 16 2020, 9:00 AM

Change 579894 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly

https://gerrit.wikimedia.org/r/579894

gerritbot added a comment.Mar 16 2020, 9:47 AM

Change 579894 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly

https://gerrit.wikimedia.org/r/579894

gerritbot added a comment.Mar 16 2020, 9:56 AM

Change 579901 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly

https://gerrit.wikimedia.org/r/579901

gerritbot added a comment.Mar 16 2020, 10:01 AM

Change 579901 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Change bacula backup frequency of dbprov to weekly

https://gerrit.wikimedia.org/r/579901

gerritbot added a comment.Mar 23 2020, 11:59 AM

Change 582791 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies

https://gerrit.wikimedia.org/r/582791

gerritbot added a comment.Mar 23 2020, 4:56 PM

Change 582791 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies

https://gerrit.wikimedia.org/r/582791

gerritbot added a comment.Mar 24 2020, 10:54 AM

Change 583049 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Stop replication for s3 when snapshotting

https://gerrit.wikimedia.org/r/583049

gerritbot added a comment.Mar 24 2020, 11:26 AM

Change 583049 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Stop replication for s3 when snapshotting

https://gerrit.wikimedia.org/r/583049

gerritbot added a comment.Mar 30 2020, 1:23 PM

Change 584599 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] partman: Reference bacula recipe (unused to prevent accidental formatting)

https://gerrit.wikimedia.org/r/584599

gerritbot added a comment.Mar 30 2020, 1:45 PM

Change 584599 merged by Jcrespo:
[operations/puppet@production] partman: Reference bacula recipe (unused to prevent accidental formatting)

https://gerrit.wikimedia.org/r/584599

gerritbot added a comment.Apr 14 2020, 10:57 AM

Change 588668 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] bacula: Increase max total size of Databases backups to 40 TB

https://gerrit.wikimedia.org/r/588668

gerritbot added a comment.Apr 15 2020, 8:48 AM

Change 588668 merged by Jcrespo:
[operations/puppet@production] bacula: Increase max total size of Databases backups to 40 TB

https://gerrit.wikimedia.org/r/588668

gerritbot added a comment.Apr 16 2020, 8:15 AM

Change 589266 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move all backup config templates to its own subdir

https://gerrit.wikimedia.org/r/589266

gerritbot added a comment.Apr 16 2020, 8:54 AM

Change 589266 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move all backup config templates to its own subdir

https://gerrit.wikimedia.org/r/589266

gerritbot added a comment.Apr 21 2020, 9:24 AM

Change 591309 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move OK alert description to wikitech

https://gerrit.wikimedia.org/r/591309

gerritbot added a comment.Apr 21 2020, 10:18 AM

Change 591309 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move OK alert description to wikitech

https://gerrit.wikimedia.org/r/591309

gerritbot added a comment.Apr 21 2020, 12:30 PM

Change 591326 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Check backups size also based on previous runs

https://gerrit.wikimedia.org/r/591326

gerritbot added a comment.Apr 27 2020, 6:55 AM

Change 591326 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Check backups size also based on previous runs

https://gerrit.wikimedia.org/r/591326

gerritbot added a comment.Apr 27 2020, 7:22 AM

Change 592599 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Tune backup size monitoring threasholds

https://gerrit.wikimedia.org/r/592599

gerritbot added a comment.Apr 27 2020, 8:11 AM

Change 592599 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Tune backup size monitoring thresholds

https://gerrit.wikimedia.org/r/592599

jcrespo updated the task description. (Show Details)Apr 27 2020, 8:20 AM
gerritbot added a comment.Apr 27 2020, 8:35 AM

Change 592608 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/592608

gerritbot added a comment.Apr 27 2020, 8:40 AM

Change 592608 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/592608

gerritbot added a comment.May 4 2020, 8:03 AM

Change 594096 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] transfer.py: Set timeout to 5 minutes

https://gerrit.wikimedia.org/r/594096

gerritbot added a comment.May 4 2020, 8:13 AM

Change 594096 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] transfer.py: Set timeout to 5 minutes

https://gerrit.wikimedia.org/r/594096

gerritbot added a comment.May 4 2020, 8:21 AM

Change 594099 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/594099

gerritbot added a comment.May 4 2020, 8:40 AM

Change 594099 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Update transfer.py to HEAD

https://gerrit.wikimedia.org/r/594099

gerritbot added a comment.May 13 2020, 11:51 AM

Change 596189 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover zarcillo from db1115 to db2093

https://gerrit.wikimedia.org/r/596189

gerritbot added a comment.May 13 2020, 12:18 PM

Change 596197 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dbtools: point zarcillo database scripts to use db2093

https://gerrit.wikimedia.org/r/596197

gerritbot added a comment.May 13 2020, 2:57 PM

Change 596189 merged by Jcrespo:
[operations/puppet@production] mariadb: Switchover zarcillo from db1115 to db2093

https://gerrit.wikimedia.org/r/596189

gerritbot added a comment.May 13 2020, 3:54 PM

Change 596197 merged by Jcrespo:
[operations/software@master] dbtools: point zarcillo database scripts to use db2093

https://gerrit.wikimedia.org/r/596197

gerritbot added a comment.May 13 2020, 3:59 PM

Change 596242 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw

https://gerrit.wikimedia.org/r/596242

gerritbot added a comment.May 13 2020, 4:02 PM

Change 596242 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: switch monitoring of tendril/zarcillo backups to codfw

https://gerrit.wikimedia.org/r/596242

gerritbot added a comment.May 13 2020, 5:23 PM

Change 596251 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Allow zarcillo as a valid backup section

https://gerrit.wikimedia.org/r/596251

gerritbot added a comment.May 13 2020, 5:24 PM

Change 596251 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Allow zarcillo as a valid backup section

https://gerrit.wikimedia.org/r/596251

gerritbot added a comment.May 18 2020, 8:15 AM

Change 597005 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Disable monitoring screens on database backup hosts

https://gerrit.wikimedia.org/r/597005

gerritbot added a comment.May 18 2020, 8:15 AM

Change 597005 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Disable monitoring screens on database backup hosts

https://gerrit.wikimedia.org/r/597005

gerritbot added a comment.Jun 26 2020, 2:13 PM

Change 608053 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move transferpy deployment to debian package

https://gerrit.wikimedia.org/r/608053

gerritbot added a comment.Jun 29 2020, 8:19 AM

Change 608053 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move transferpy deployment to debian package

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608053

gerritbot added a comment.Jul 2 2020, 7:45 AM

Change 609101 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Fix remote_backup_mariadb.py deps and path

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609101

gerritbot added a comment.Jul 2 2020, 7:48 AM

Change 609101 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Fix remote_backup_mariadb.py deps and path

https://gerrit.wikimedia.org/r/c/operations/puppet/ /609101

gerritbot added a comment.Tue, Jul 21, 9:02 AM

Change 615155 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Adjust check parameters to get less false positives

https://gerrit.wikimedia.org/r/615155

gerritbot added a comment.Tue, Jul 21, 9:18 AM

Change 615155 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Adjust check parameters to get less false positives

https://gerrit.wikimedia.org/r/615155

gerritbot added a comment.Thu, Aug 6, 9:20 AM

Change 618720 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Reenable notifications for dbprov2003 after maintenance

https://gerrit.wikimedia.org/r/618720

gerritbot added a comment.Thu, Aug 6, 9:40 AM

Change 618722 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move x1 and misc logical dumps to dbprov1003

https://gerrit.wikimedia.org/r/618722

gerritbot added a comment.Thu, Aug 6, 9:47 AM

Change 618723 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] BackupStatistics: Do not raise an exception if metadata cannot be sent

https://gerrit.wikimedia.org/r/618723

gerritbot added a comment.Thu, Aug 6, 9:53 AM

Change 618723 abandoned by Jcrespo:
[operations/software/wmfmariadbpy@master] BackupStatistics: Do not raise an exception if metadata cannot be sent

Reason:
not a bug

https://gerrit.wikimedia.org/r/618723

gerritbot added a comment.Thu, Aug 6, 10:47 AM

Change 618720 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Reenable notifications for dbprov2003 after maintenance

https://gerrit.wikimedia.org/r/618720

gerritbot added a comment.Thu, Aug 6, 1:49 PM

Change 618722 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move x1 and misc logical dumps to dbprov1003

https://gerrit.wikimedia.org/r/618722

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL