Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Kormat
	May 29 2020, 7:12 AM

Description

wmfbackups.WMFBackup (xtrabackup option) runs prepare on a given path:

https://phabricator.wikimedia.org/diffusion/OSWB/browse/master/wmfbackups/MariaBackup.py$58

However, it could be that, accidentally, we run prepare on an xtrabackup version that is different that that of the original mariadb.

Ideally, the version would be the same. It is ok if it is the same major version (10.1, 10.4), but a higher minor version. It is undesired if it is the same major version but a lower version minor version. It should prevent running with a different major version (and specially if it is a lower one).

There is also the chance of running xtrabackup from Percona on a MariaDB server dump. This should also be prevented. Checking the server version that generated the 'xtrabackup_info' has a server_version and comparing it to the xtrabackup version could be enough- and log a nice error message.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
mariabackup: Make the vendor detection account for known variations	operations/software/wmfbackups	master	+63 -7
Check for server version and compare with xtrabackup	operations/software/wmfbackups	master	+13 -0
Check that xtrabackup --prepare is using the same version	operations/software/wmfbackups	master	+512 -57

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	jcrespo	T274636 Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects
Open	None	T138562 Improve regular production database backups handling
Resolved	jcrespo	T253959 Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken

Event Timeline

• Kormat created this task.May 29 2020, 7:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 29 2020, 7:12 AM

Marostegui triaged this task as Medium priority.May 29 2020, 12:10 PM

Marostegui moved this task from Triage to Backlog on the DBA board.

LSobanski edited projects, added Data-Persistence-Backup; removed DBA.Oct 6 2020, 9:20 AM

jcrespo moved this task from Triage to Refine on the Data-Persistence-Backup board.Dec 21 2020, 10:35 PM

Due to my comments on https://gerrit.wikimedia.org/r/599343 I think this is something we could improve from wmfbackups, but it is not a high priority for now (should not be a regular concern, but some extra automation could be setup, if we think of a way).

RhinosF1 subscribed.Mar 25 2021, 10:06 AM

jcrespo renamed this task from Care needed with mariabackup versions to Check we are preparing (xtrabackup --prepare) with the same package version than the backup was taken.Mar 25 2021, 10:14 AM

jcrespo renamed this task from Check we are preparing (xtrabackup --prepare) with the same package version than the backup was taken to Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken.

jcrespo edited projects, added good first task; removed Data-Persistence-Backup.

jcrespo updated the task description. (Show Details)

jcrespo added a parent task: T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.

jcrespo mentioned this in T274636: Develop a web dashboard or a command line tool to help inventory and/or monitor database and backup objects.Mar 25 2021, 10:17 AM

@jcrespo, I would like to try my hand on this. I have cloned the repo and tried to build this package
But it fails at the end

dpkg-source --after-build .
dpkg-source: info: using options from wmfbackups/debian/source/options: --extend-diff-ignore=^[^/]*[.]egg-info/ --tar-ignore=.*
dpkg-buildpackage: info: binary-only upload (no source included)
Now running lintian wmfbackups_0.4+deb10u2_amd64.changes ...
E: wmfbackups changes: bad-distribution-in-changes-file buster
W: wmfbackups: binary-without-manpage usr/bin/backup-mariadb
W: wmfbackups: binary-without-manpage usr/bin/recover-dump
W: wmfbackups-check: binary-without-manpage usr/bin/check-mariadb-backups
W: wmfbackups-remote: binary-without-manpage usr/bin/remote-backup-mariadb
Finished running lintian.

Please guide me on it. :)

No issue, but I would prefer to focus on having the first 2 patches you proposed merged first. It is about quality of patches, not quantity. You (like other candidates) should start thinking about creating a phabricator proposal draft -not final- now https://www.mediawiki.org/wiki/Google_Summer_of_Code/Participants#Application_process_steps (step 10).

You (like other candidates) should start thinking about creating a phabricator proposal draft -not final- now https://www.mediawiki.org/wiki/Google_Summer_of_Code/Participants#Application_process_steps (step 10).

Okay sure, thankyou for suggesting, I'll start working on it. I added a comment on my previous patch too, please review whenever you get time.

hi @jcrespo, meanwhile my other patches are getting reviewed, may I continue working on this task? I need some guidance on building the package as I mentioned earlier :)

jcrespo added a project: dbbackups-dashboard.Apr 6 2021, 6:55 AM

@Palak199 Nobody else expressed interest on it so far, so I guess it is ok.

Building the package is not important; as long as the existing test pass, it will be mergable. It says "bad-distribution-in-changes-file buster"- maybe you are building on ubuntu or old version of debian? buster should be a valid distro name, and it works for me. If it is the only thing failing, you should not worry about it.

In T253959#6975023, @jcrespo wrote:

@Palak199 Nobody else expressed interest on it so far, so I guess it is ok.

Sure ! I am doing it.

It says "bad-distribution-in-changes-file buster"- maybe you are building on ubuntu or old version of debian?

I am building it on 20.04 Ubuntu focal fossa
But if building is not important, I'll leave it and try doing the task and test it.
it might sound childish but I am not sure, how am I supposed to get these two versions? and also how would I know if it is percona?

• Kormat unsubscribed.Apr 8 2021, 8:13 AM

Hi @jcrespo,
I researched and found out that the xtrabackup_info itself has the server version. I think we can get the xtrabackup version from there.
And regarding the server version, I am doubtful. Can we use the pymysql to query that?

I researched and found out that the xtrabackup_info itself has the server version. I think we can get the xtrabackup version from there

Indeed that is the right way.

And regarding the server version, I am doubtful. Can we use the pymysql to query that?

I am confused about this part- if you get the server version from the file, you won't need it again from the backed up server, right? Or maybe you mean the place where you run "--prepare"? Note if the latter, there is likely not to be a running server there, as in particular, for our setup, we send the backups to a remote location with just the binaries and no running server (we run --prepare just after backup, not before recovery). In this case, a mysql connection may not be the best place, and maybe using something like "xtrabackup --version" could be thought instead?

I am building it on 20.04 Ubuntu focal fossa

Then that is the source of the issue- you can try to put there an ubuntu distro name and build for it, but definitely not necessary. You should be able to work on all other tasks on ubuntu, given the similarity of environments. FYI, we will aim for a working Buster result in most of our tasks, as that is what runs on Production Wikimedia.

In this case, a mysql connection may not be the best place, and maybe using something like "xtrabackup --version" could be thought instead?

in this xtrabackup_info file
Am i supposed to match the ibbackup_version(here) to the server_version(here)?

In T253959#6987450, @jcrespo wrote:

I am building it on 20.04 Ubuntu focal fossa

FYI, we will aim for a working Buster result in most of our tasks, as that is what runs on Production Wikimedia.

Okay! I'll switch to debian when working with Wikimedia projects :)

Unless I am not mistaken (we should test it), it would be the "server_version" from the generated file, to the "xtrabackup --version" standard output from the place where "--prepare" runs.

Note we use mariabackup on production, but we assume there will be an xtrabackup executable on path. E.g. on a regular server:

jynus:~$ sudo apt install mariadb-backup
[sudo] password for jynus: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  mariadb-backup
0 upgraded, 1 newly installed, 0 to remove and 50 not upgraded.
Need to get 6,050 kB of archives.
After this operation, 27.6 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 mariadb-backup amd64 1:10.5.9-1 [6,050 kB]
Fetched 6,050 kB in 0s (18.3 MB/s)   
Selecting previously unselected package mariadb-backup.
(Reading database ... 248135 files and directories currently installed.)
Preparing to unpack .../mariadb-backup_1%3a10.5.9-1_amd64.deb ...
Unpacking mariadb-backup (1:10.5.9-1) ...
Setting up mariadb-backup (1:10.5.9-1) ...
Processing triggers for man-db (2.9.4-2) ...

jynus@~$ mariabackup --version
mariabackup based on MariaDB server 10.5.9-MariaDB debian-linux-gnu (x86_64)

On production:

dbprov1001:~$ xtrabackup --version
xtrabackup based on MariaDB server 10.1.44-MariaDB Linux (x86_64) 
dbprov1001:~$ ls -lha $(which xtrabackup)
lrwxrwxrwx 1 root staff 28 Aug 17  2020 /usr/local/bin/xtrabackup -> /etc/alternatives/xtrabackup
dbprov1001:~$ update-alternatives --list xtrabackup
/opt/wmf-mariadb101/bin/mariabackup

In this case, for production, if we had a 10.4 backup and wanted to prepare with a 10.1 xtrabackup/mariabackup executable, we should probably throw an error instead.

As I said, this is probably the most complex microtask, but once you navigate the complexities, probably the easiest to implement (a condition + error).

Change 678299 had a related patch set uploaded (by Palak199; author: Palak199):

[operations/software/wmfbackups@master] Check for server version and compare with xtrabackup

https://gerrit.wikimedia.org/r/678299

gerritbot added a project: Patch-For-Review.Apr 10 2021, 4:13 AM

@jcrespo

[operations/software/wmfbackups@master] Check for server version and compare with xtrabackup

https://gerrit.wikimedia.org/r/678299

In this patch I have just started coding what I understood from the above messages.
Currently,

I have checked the full version and thrown error for anomaly be it in major or minor version, this I'll improve soon.

(By logging a warning for minor version inconsistency and error on major version inconsistency.)

I have added the check in errors_on_metadata() function as the metadata file was already being read there

Let me know if this check should have its own function.

P.S. just a baby step towards the solution

I will be checking this contribution soon as part of my job maintaining backups, and hopefully, getting it merged.

jcrespo moved this task from Refine to Ready on the Data-Persistence-Backup board.Jun 1 2021, 8:42 AM

jcrespo mentioned this in T284157: Use mariabackup instead of xtrabackup for galera backups? (Or possibly for all maria backups?).Jun 2 2021, 3:52 PM

jcrespo added a project: database-backups.Nov 17 2021, 11:35 AM

jcrespo added a parent task: T138562: Improve regular production database backups handling.

jcrespo removed a project: good first task.Mar 8 2022, 10:34 AM

Requested the logic to be implemented upstream: https://jira.mariadb.org/browse/MDEV-23718

Change 769428 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/wmfbackups@master] Check that xtrabackup --prepare is using the same version

https://gerrit.wikimedia.org/r/769428

Change 769428 merged by Jcrespo:

[operations/software/wmfbackups@master] Check that xtrabackup --prepare is using the same version

https://gerrit.wikimedia.org/r/769428

Change 678299 abandoned by Jcrespo:

[operations/software/wmfbackups@master] Check for server version and compare with xtrabackup

Reason:

Implemented at https://gerrit.wikimedia.org/r/c/operations/software/wmfbackups/+/769428 with tests

https://gerrit.wikimedia.org/r/678299

A fix has been merged- waiting only to get a newer release to deploy the change.

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2022, 5:10 PM

Change deployed.

jcrespo mentioned this in T293975: Document troubleshooting a failing snapshot backup.Mar 18 2022, 7:16 PM

Change 800708 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/wmfbackups@master] mariabackup: Make the vendor detection account for known variations

https://gerrit.wikimedia.org/r/800708

gerritbot added a project: Patch-For-Review.May 27 2022, 11:15 AM

Change 800708 merged by Jcrespo: