Page MenuHomePhabricator

[toolsdb] Set up tools-db-2 to replicate tools-db-1
Closed, ResolvedPublic

Description

tools-db-2 will become the new replica after tools-db-1 is promoted to primary.

We can set up chain replication from tools-db-1, without having to wait for it to be a primary. To start the replication, I think we might be able to leverage Cinder snapshots, creating a snapshot of the Cinder volume in tools-db-1, then creating a new volume from that snapshot and attaching it to tools-db-2.

Event Timeline

fnegri changed the task status from Open to In Progress.Feb 13 2023, 6:34 PM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2022/2023-Q3) board.
fnegri triaged this task as High priority.Mar 6 2023, 8:58 AM
  • Created a Cinder snapshot of the tools-db-1 volume using the Horizon web interface
  • Created a new Cinder volume tools-db-2, based on that snapshot
  • Attached the new volume to the tools-db-2 instance
  • Deleted all the binlogs from /srv/labsdb/tmp (probably not required, but to avoid having old files around in the new instance)
  • Started mariadb with systemctl start mariadb
  • Set the new instance to replicate from tools-db-1:
MariaDB [(none)]> RESET SLAVE;
MariaDB [(none)]> CHANGE MASTER TO  MASTER_HOST='tools-db-1.tools.eqiad1.wikimedia.cloud', MASTER_USER='repl',  MASTER_PASSWORD='xxxxxx', MASTER_USE_GTID=slave_pos;
MariaDB [(none)]> START SLAVE;

tools-db-2 is replicating correctly from tools-db-1 and is now in sync.

Screenshot 2023-03-09 at 18.01.21.png (952×1 px, 134 KB)