[toolsdb] Set up tools-db-2 to replicate tools-db-1
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fnegri
	Feb 13 2023, 2:20 PM

Description

tools-db-2 will become the new replica after tools-db-1 is promoted to primary.

We can set up chain replication from tools-db-1, without having to wait for it to be a primary. To start the replication, I think we might be able to leverage Cinder snapshots, creating a snapshot of the Cinder volume in tools-db-1, then creating a new volume from that snapshot and attaching it to tools-db-2.

Related Objects
Search...

Status	Assigned	Task
Resolved	• Bstorm	T216208 ToolsDB overload and cleanup
Resolved	• Bstorm	T216441 Evaluate transferring the non-replicated tables to the new toolsdb server
Resolved	fnegri	T236101 Find a way to remove non-replicated tables from ToolsDB
Resolved	None	T306061 Cloud VPS "clouddb-services" project Stretch deprecation
Resolved	None	T313984 cloudvirt1019: hpssacli not found
Resolved	fnegri	T301949 ToolsDB upgrade => Bullseye, MariaDB 10.4
Resolved	fnegri	T329521 [toolsdb] Set up tools-db-2 to replicate tools-db-1
Resolved	fnegri	T329970 [toolsdb] set up tools-db-1 to replicate from clouddb1001
Resolved	fnegri	T330655 monitoring: Disk space check can fail to read fuse mounts

Event Timeline

fnegri created this task.Feb 13 2023, 2:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2023, 2:20 PM

fnegri mentioned this in T301949: ToolsDB upgrade => Bullseye, MariaDB 10.4.Feb 13 2023, 2:28 PM

fnegri changed the task status from Open to In Progress.Feb 13 2023, 6:34 PM

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2022/2023-Q3) board.

fnegri added a subtask: T329970: [toolsdb] set up tools-db-1 to replicate from clouddb1001.Feb 17 2023, 6:13 PM

fnegri changed the status of subtask T329970: [toolsdb] set up tools-db-1 to replicate from clouddb1001 from Open to In Progress.Feb 21 2023, 3:49 PM

fnegri triaged this task as High priority.Mar 6 2023, 8:58 AM

fnegri closed subtask T329970: [toolsdb] set up tools-db-1 to replicate from clouddb1001 as Resolved.Mar 9 2023, 12:11 PM

fnegri reopened subtask T329970: [toolsdb] set up tools-db-1 to replicate from clouddb1001 as In Progress.Mar 9 2023, 1:55 PM

Created a Cinder snapshot of the tools-db-1 volume using the Horizon web interface
Created a new Cinder volume tools-db-2, based on that snapshot
Attached the new volume to the tools-db-2 instance
Deleted all the binlogs from /srv/labsdb/tmp (probably not required, but to avoid having old files around in the new instance)
Started mariadb with systemctl start mariadb
Set the new instance to replicate from tools-db-1:

MariaDB [(none)]> RESET SLAVE;
MariaDB [(none)]> CHANGE MASTER TO  MASTER_HOST='tools-db-1.tools.eqiad1.wikimedia.cloud', MASTER_USER='repl',  MASTER_PASSWORD='xxxxxx', MASTER_USE_GTID=slave_pos;
MariaDB [(none)]> START SLAVE;

fnegri closed subtask T329970: [toolsdb] set up tools-db-1 to replicate from clouddb1001 as Resolved.Mar 9 2023, 3:59 PM

tools-db-2 is replicating correctly from tools-db-1 and is now in sync.