At very least, it is clear that Galera SST processes are supported by mariabackup. I'm not sure about xtrabackup, but it might be.

the WMF package tries to standarize this for future extensibility. This is just FYI, as you probably use a different package and can organize things differently, but it may help you to do the same or see how we do it.

wmf-mariadb104 creates its own binary, called mariabackup.

One of the changes we do on our package is to create a new alternative name on install for xtrabackup- each wmf-like package provides xtrabackup, but it can be providede by mariadb, percona, mysql, etc. That way you can chose what is run when doing xtrabackup, which version is exposed by default.

SST normally allow to configure the actual command executed.

If you use mysql packages, you should use mysql xtrabackup compiled preferibly with the same version, and compulsory, with the same or higher of the same vendor.
Same for mariadb. Percona is, for the most part, interchangable with mysql (unless using some of the extra features that are only on percona).

If you try to backup mysql with a mariadb-compiled one or the other way round, since 10.0, it will fail with an error.

The backup has a pending feature to check versions are compatible (which is enforced now statically on configuration, but we want to also check it at run time to avoid issues): T253959

Looks like for cloud (since we are using upstream), we should use https://packages.debian.org/hu/buster/mariadb-backup

Not sure what the puppet bacula thing is doing :)

Change 698064 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] galera: ensure that mariabackup is also installed

https://gerrit.wikimedia.org/r/698064

If I'm reading what @jcrespo said correctly, it would probably be wise for us to have mariabackup installed on our galera cluster since our setup won't have the wmf-compiled binary (see patch).

As for the bacula classes, it seems very likely that it is doing the right things under the hood, but restore testing is never a bad idea. We could even restore some tables to a test server to be sure.

After considerable code-diving, we have determined that we aren't using mariabackup OR xtrabackup for this but rather mysqldump.

That's probably fine -- these are small databases and probably always will be, and I appreciate the human-readableness of mysqldump.

So that leaves us with only one real followup question here (in case @jcrespo is still following along): these backups are currently going into the 'production' bacula pool. Is that the right pool?

Change 698064 merged by Bstorm:

[operations/puppet@production] galera: ensure that mariabackup is also installed

https://gerrit.wikimedia.org/r/698064

Change 698251 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] galera: clear up confusing xtrabackup parameter

https://gerrit.wikimedia.org/r/698251

In T284157#7135507, @Andrew wrote:

So that leaves us with only one real followup question here (in case @jcrespo is still following along): these backups are currently going into the 'production' bacula pool. Is that the right pool?

Gray area-- let's talk.

get rid of the confusing xtrabackup parameter unless we actually start using it with the bpipe method

We are trying to deprecate that method, as it causes issue for bacula scheduling. I believe the host you are intended to backup are production (even if production-labs-support), in that case it would be very easy to add new hosts to the existing dbbackups workflow (even using exclusively your own hardware). The backup process should be compatible with galera and other packages, although may require some testing.

Let's also talk- the reason why we redid the database backups on production was because the previous method (the one you are using / intend to use) had huge architectural flaws- although we didn't push for non-persistence-teams to convert to the new method (avoiding the slow mysqldump, and enbracing modern tools such as mydumper and xtrabackup/mariabackup).

Thanks for your time today, please continue with the work under the scope of this ticket at your own pace- and I will create a new ticket for database backups workflow migration for cloud dbs.

Change 698251 merged by Bstorm:

[operations/puppet@production] galera: clear up confusing xtrabackup parameter

https://gerrit.wikimedia.org/r/698251