Page MenuHomePhabricator

Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken
Open, LowPublic

Description

wmfbackups.WMFBackup (xtrabackup option) runs prepare on a given path:

https://phabricator.wikimedia.org/diffusion/OSWB/browse/master/wmfbackups/MariaBackup.py$58

However, it could be that, accidentally, we run prepare on an xtrabackup version that is different that that of the original mariadb.

Ideally, the version would be the same. It is ok if it is the same major version (10.1, 10.4), but a higher minor version. It is undesired if it is the same major version but a lower version minor version. It should prevent running with a different major version (and specially if it is a lower one).

There is also the chance of running xtrabackup from Percona on a MariaDB server dump. This should also be prevented. Checking the server version that generated the 'xtrabackup_info' has a server_version and comparing it to the xtrabackup version could be enough- and log a nice error message.

Event Timeline

Marostegui moved this task from Triage to Backlog on the DBA board.
jcrespo lowered the priority of this task from Medium to Low.Mar 25 2021, 10:05 AM
jcrespo added a subscriber: jcrespo.

Due to my comments on https://gerrit.wikimedia.org/r/599343 I think this is something we could improve from wmfbackups, but it is not a high priority for now (should not be a regular concern, but some extra automation could be setup, if we think of a way).

jcrespo renamed this task from Care needed with mariabackup versions to Check we are preparing (xtrabackup --prepare) with the same package version than the backup was taken.Mar 25 2021, 10:14 AM
jcrespo renamed this task from Check we are preparing (xtrabackup --prepare) with the same package version than the backup was taken to Check we are preparing (xtrabackup --prepare) with the same package version as the server version of which the backup was taken.
jcrespo edited projects, added good first task; removed Data-Persistence-Backup.
jcrespo updated the task description. (Show Details)

@jcrespo, I would like to try my hand on this. I have cloned the repo and tried to build this package
But it fails at the end

dpkg-source --after-build .
dpkg-source: info: using options from wmfbackups/debian/source/options: --extend-diff-ignore=^[^/]*[.]egg-info/ --tar-ignore=.*
dpkg-buildpackage: info: binary-only upload (no source included)
Now running lintian wmfbackups_0.4+deb10u2_amd64.changes ...
E: wmfbackups changes: bad-distribution-in-changes-file buster
W: wmfbackups: binary-without-manpage usr/bin/backup-mariadb
W: wmfbackups: binary-without-manpage usr/bin/recover-dump
W: wmfbackups-check: binary-without-manpage usr/bin/check-mariadb-backups
W: wmfbackups-remote: binary-without-manpage usr/bin/remote-backup-mariadb
Finished running lintian.

Please guide me on it. :)

No issue, but I would prefer to focus on having the first 2 patches you proposed merged first. It is about quality of patches, not quantity. You (like other candidates) should start thinking about creating a phabricator proposal draft -not final- now https://www.mediawiki.org/wiki/Google_Summer_of_Code/Participants#Application_process_steps (step 10).

You (like other candidates) should start thinking about creating a phabricator proposal draft -not final- now https://www.mediawiki.org/wiki/Google_Summer_of_Code/Participants#Application_process_steps (step 10).

Okay sure, thankyou for suggesting, I'll start working on it. I added a comment on my previous patch too, please review whenever you get time.

hi @jcrespo, meanwhile my other patches are getting reviewed, may I continue working on this task? I need some guidance on building the package as I mentioned earlier :)

@Palak199 Nobody else expressed interest on it so far, so I guess it is ok.

Building the package is not important; as long as the existing test pass, it will be mergable. It says "bad-distribution-in-changes-file buster"- maybe you are building on ubuntu or old version of debian? buster should be a valid distro name, and it works for me. If it is the only thing failing, you should not worry about it.

@Palak199 Nobody else expressed interest on it so far, so I guess it is ok.

Sure ! I am doing it.

It says "bad-distribution-in-changes-file buster"- maybe you are building on ubuntu or old version of debian?

I am building it on 20.04 Ubuntu focal fossa
But if building is not important, I'll leave it and try doing the task and test it.
it might sound childish but I am not sure, how am I supposed to get these two versions? and also how would I know if it is percona?

Hi @jcrespo,
I researched and found out that the xtrabackup_info itself has the server version. I think we can get the xtrabackup version from there.
And regarding the server version, I am doubtful. Can we use the pymysql to query that?

I researched and found out that the xtrabackup_info itself has the server version. I think we can get the xtrabackup version from there

Indeed that is the right way.

And regarding the server version, I am doubtful. Can we use the pymysql to query that?

I am confused about this part- if you get the server version from the file, you won't need it again from the backed up server, right? Or maybe you mean the place where you run "--prepare"? Note if the latter, there is likely not to be a running server there, as in particular, for our setup, we send the backups to a remote location with just the binaries and no running server (we run --prepare just after backup, not before recovery). In this case, a mysql connection may not be the best place, and maybe using something like "xtrabackup --version" could be thought instead?

I am building it on 20.04 Ubuntu focal fossa

Then that is the source of the issue- you can try to put there an ubuntu distro name and build for it, but definitely not necessary. You should be able to work on all other tasks on ubuntu, given the similarity of environments. FYI, we will aim for a working Buster result in most of our tasks, as that is what runs on Production Wikimedia.

In this case, a mysql connection may not be the best place, and maybe using something like "xtrabackup --version" could be thought instead?

image.png (313×625 px, 33 KB)

in this xtrabackup_info file
Am i supposed to match the ibbackup_version(here) to the server_version(here)?

I am building it on 20.04 Ubuntu focal fossa

FYI, we will aim for a working Buster result in most of our tasks, as that is what runs on Production Wikimedia.

Okay! I'll switch to debian when working with Wikimedia projects :)

Unless I am not mistaken (we should test it), it would be the "server_version" from the generated file, to the "xtrabackup --version" standard output from the place where "--prepare" runs.

Note we use mariabackup on production, but we assume there will be an xtrabackup executable on path. E.g. on a regular server:

jynus:~$ sudo apt install mariadb-backup
[sudo] password for jynus: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  mariadb-backup
0 upgraded, 1 newly installed, 0 to remove and 50 not upgraded.
Need to get 6,050 kB of archives.
After this operation, 27.6 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 mariadb-backup amd64 1:10.5.9-1 [6,050 kB]
Fetched 6,050 kB in 0s (18.3 MB/s)   
Selecting previously unselected package mariadb-backup.
(Reading database ... 248135 files and directories currently installed.)
Preparing to unpack .../mariadb-backup_1%3a10.5.9-1_amd64.deb ...
Unpacking mariadb-backup (1:10.5.9-1) ...
Setting up mariadb-backup (1:10.5.9-1) ...
Processing triggers for man-db (2.9.4-2) ...

jynus@~$ mariabackup --version
mariabackup based on MariaDB server 10.5.9-MariaDB debian-linux-gnu (x86_64)

On production:

dbprov1001:~$ xtrabackup --version
xtrabackup based on MariaDB server 10.1.44-MariaDB Linux (x86_64) 
dbprov1001:~$ ls -lha $(which xtrabackup)
lrwxrwxrwx 1 root staff 28 Aug 17  2020 /usr/local/bin/xtrabackup -> /etc/alternatives/xtrabackup
dbprov1001:~$ update-alternatives --list xtrabackup
/opt/wmf-mariadb101/bin/mariabackup

In this case, for production, if we had a 10.4 backup and wanted to prepare with a 10.1 xtrabackup/mariabackup executable, we should probably throw an error instead.

As I said, this is probably the most complex microtask, but once you navigate the complexities, probably the easiest to implement (a condition + error).

Change 678299 had a related patch set uploaded (by Palak199; author: Palak199):

[operations/software/wmfbackups@master] Check for server version and compare with xtrabackup

https://gerrit.wikimedia.org/r/678299

@jcrespo

[operations/software/wmfbackups@master] Check for server version and compare with xtrabackup

https://gerrit.wikimedia.org/r/678299

In this patch I have just started coding what I understood from the above messages.
Currently,

  • I have checked the full version and thrown error for anomaly be it in major or minor version, this I'll improve soon.

(By logging a warning for minor version inconsistency and error on major version inconsistency.)

  • I have added the check in errors_on_metadata() function as the metadata file was already being read there

Let me know if this check should have its own function.

P.S. just a baby step towards the solution

I will be checking this contribution soon as part of my job maintaining backups, and hopefully, getting it merged.