Page MenuHomePhabricator

Make a block-level copy of the codfw mirror of labstore1001 to eqiad
Closed, ResolvedPublic

Description

Which will be resynced from the live system locally afterwards.

Event Timeline

coren created this task.Jun 1 2015, 5:59 PM
coren claimed this task.
coren raised the priority of this task from to High.
coren updated the task description. (Show Details)
coren added subscribers: mark, faidon, Ltrlg and 5 others.

First part is having the destination ready, doing that.

coren moved this task from To Do to Doing on the Labs-Sprint-100 board.Jun 1 2015, 6:05 PM
coren added a comment.EditedJun 1 2015, 6:56 PM

Destination volume on labstore1002 is set up

backup/backup aka /dev/mapper/backup-backup

It has 40T virtual size (as the source volume in codfw) and 16T of actual pool space behind it (which should suffice, as there is a bit under 10T of data to back up).

As a reminder, no volumes in the vg "space" should be touched in any way as the controller it lives on has proven unreliable and has not been debugged.

coren reassigned this task from coren to mark.Jun 1 2015, 6:58 PM
mark added a comment.EditedJun 2 2015, 8:56 AM

(Sparse) block level copying of the thin volumes started between the systems:

pv -eprab /dev/mapper/store-now_snap | pigz -cf - | ssh -o Compression=no -o CompressionLevel=0 root@labstore1002.eqiad.wmnet 'unpigz -c | dd of=/dev/mapper/backup-backup conv=sparse' bs=4k
coren added a comment.Jun 3 2015, 1:20 PM

The copy is progressing nicely (if not as fast as hoped); the bottleneck appears to be the ssh channel window size rather than actual bandwidth.

coren added a comment.Jun 4 2015, 1:10 PM

The copy had to be restarted; for some reason, the amount of metadata at the destination appeared to be bigger than that needed at the source (perhaps a simple matter of different extent size given the different backing store).

coren closed this task as Resolved.Jun 8 2015, 12:41 PM

The copy is complete, and is mounted at the destination.

A caveat worth nothing: since the source filesystem uses an external journal and that journal is (very explicitly) not copied along, the destination must be mounted without journal (or, as I have done, the journal needs to be removed). This is, of course, completely okay for a backup filesystem.

tune2fs -f -O ^has_journal $filesystem
coren moved this task from Doing to Done on the Labs-Sprint-100 board.Jun 8 2015, 12:42 PM