Page MenuHomePhabricator

[quarry] Move quarry database to Trove, update backup system
Closed, ResolvedPublic

Description

  1. Write the description below

Event Timeline

dcaro triaged this task as High priority.Aug 24 2021, 11:44 AM
dcaro created this task.

Moved the new (buster) hosts to use the trove DB instance, I had to remove a non-longer use column from the
query_revisions table (dbs was replaced by query_database at some point, but it's still left over on the old database).

Left it running there.

@Mdipietro Currently the buster deployment is completely separate from the old one, as they are using different DBs and
redis instances (see T290059).

Bstorm subscribed.

Just grabbing this to slap my method of backing up from paws in here with adjustments now that we are on trove. We can revisit later when we have native trove backups (swift).

Change 715997 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] quarry: add a simple backup server

https://gerrit.wikimedia.org/r/715997

Mentioned in SAL (#wikimedia-cloud) [2021-09-01T18:26:36Z] <bstorm> started instance quarry-dbbackup-01 T289568

Change 715997 merged by Bstorm:

[operations/puppet@production] quarry: add a simple backup server

https://gerrit.wikimedia.org/r/715997

Change 716053 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] quarry backup: change the cleanup job to check number of backups

https://gerrit.wikimedia.org/r/716053

Change 716053 merged by Bstorm:

[operations/puppet@production] quarry backup: change the cleanup job to check number of backups

https://gerrit.wikimedia.org/r/716053

Part of this is working well. However, I do see an issue:

Sep  6 23:23:05 quarry-dbbackup-01 systemd[1]: quarry-cleanup-mysqldump.service: Failed with result 'exit-code'.

Change 719301 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] quarry dbbackup: fix the script typo

https://gerrit.wikimedia.org/r/719301

Change 719301 merged by Bstorm:

[operations/puppet@production] quarry dbbackup: fix the script typo

https://gerrit.wikimedia.org/r/719301

Everything looks good now.