Page MenuHomePhabricator

CloudVPS: codfw1dev: database backup for clouddb2001-dev.codfw.wmnet
Open, NormalPublic

Description

As far as I know, the clouddb2001-dev.codfw.wmnet is not getting any backup or replication or anything that would allow us to recover from a data disaster.
This was not urgent at first since this is mostly the database for a non-customer-facing openstack deployment. But anyway, having backups of some sorts would save us a lot of work/time in the case we need to recover from a outage.
I suggest we evaluate the situation and decide what do we want to do here.

I may be wrong and there is some automagical mechanism for backups that I'm not aware of.
CC'ing @Marostegui and @jcrespo for advice.

Event Timeline

aborrero triaged this task as Normal priority.Aug 1 2019, 9:14 AM
aborrero created this task.
aborrero moved this task from Inbox to Important on the cloud-services-team (Kanban) board.

We are not backuping anything related to that host, I didn't even know that host existed :-)
Given that the data is tiny:

root@clouddb2001-dev:/var/lib/mysql# du -sh .
5.9G	.

You can probably set up a cronjob running a mysqldump and placing it somewhere you trust to get those databases backed-up.

We do backup all the databases in db1073 (m5) which look similar to these, but I guess this is just development and the other one is production, right?

bd808 added a comment.Aug 2 2019, 11:38 PM

We do backup all the databases in db1073 (m5) which look similar to these, but I guess this is just development and the other one is production, right?

Correct. This is a "staging" cluster for OpenStack changes. The database here is one that we had hopes to put into T218570: DB planning: include a writeable (?) misc DB cluster in codfw for WMCS.

bd808 added a comment.Aug 2 2019, 11:48 PM

Backups could be sent to the new cloudbackup2xxx hosts once they are actually live (T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet). There might be somewhere on the current labstore2xxx hosts to put the data in the meantime.