Page MenuHomePhabricator

Make a block-level copy of the codfw mirror of labstore1001 to eqiad
Closed, ResolvedPublic


Which will be resynced from the live system locally afterwards.

Event Timeline

coren claimed this task.
coren raised the priority of this task from to High.
coren updated the task description. (Show Details)
coren added subscribers: mark, faidon, Ltrlg and 5 others.

First part is having the destination ready, doing that.

Destination volume on labstore1002 is set up

backup/backup aka /dev/mapper/backup-backup

It has 40T virtual size (as the source volume in codfw) and 16T of actual pool space behind it (which should suffice, as there is a bit under 10T of data to back up).

As a reminder, no volumes in the vg "space" should be touched in any way as the controller it lives on has proven unreliable and has not been debugged.

(Sparse) block level copying of the thin volumes started between the systems:

pv -eprab /dev/mapper/store-now_snap | pigz -cf - | ssh -o Compression=no -o CompressionLevel=0 root@labstore1002.eqiad.wmnet 'unpigz -c | dd of=/dev/mapper/backup-backup conv=sparse' bs=4k

The copy is progressing nicely (if not as fast as hoped); the bottleneck appears to be the ssh channel window size rather than actual bandwidth.

The copy had to be restarted; for some reason, the amount of metadata at the destination appeared to be bigger than that needed at the source (perhaps a simple matter of different extent size given the different backing store).

The copy is complete, and is mounted at the destination.

A caveat worth nothing: since the source filesystem uses an external journal and that journal is (very explicitly) not copied along, the destination must be mounted without journal (or, as I have done, the journal needs to be removed). This is, of course, completely okay for a backup filesystem.

tune2fs -f -O ^has_journal $filesystem