Which will be resynced from the live system locally afterwards.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | yuvipanda | T105720 Labs team reliability goal for Q1 2015/16 | |||
| Resolved | coren | T106479 Ensure that labstore machine is 'known good' hardware | |||
| Resolved | • chasemp | T98183 labstore1002 issues while trying to reboot | |||
| Resolved | • chasemp | T101741 Locate and assign some MD1200 shelves for proper testing of labstore1002 | |||
| Resolved | coren | T96063 Migrate Labs NFS storage from RAID6 to RAID10 | |||
| Resolved | coren | T101011 Rsync live labstore filesystem to local eqiad copy | |||
| Resolved | mark | T101010 Make a block-level copy of the codfw mirror of labstore1001 to eqiad |
Event Timeline
Comment Actions
Destination volume on labstore1002 is set up
backup/backup aka /dev/mapper/backup-backup
It has 40T virtual size (as the source volume in codfw) and 16T of actual pool space behind it (which should suffice, as there is a bit under 10T of data to back up).
As a reminder, no volumes in the vg "space" should be touched in any way as the controller it lives on has proven unreliable and has not been debugged.
Comment Actions
(Sparse) block level copying of the thin volumes started between the systems:
pv -eprab /dev/mapper/store-now_snap | pigz -cf - | ssh -o Compression=no -o CompressionLevel=0 root@labstore1002.eqiad.wmnet 'unpigz -c | dd of=/dev/mapper/backup-backup conv=sparse' bs=4k
Comment Actions
The copy is progressing nicely (if not as fast as hoped); the bottleneck appears to be the ssh channel window size rather than actual bandwidth.
Comment Actions
The copy had to be restarted; for some reason, the amount of metadata at the destination appeared to be bigger than that needed at the source (perhaps a simple matter of different extent size given the different backing store).
Comment Actions
The copy is complete, and is mounted at the destination.
A caveat worth nothing: since the source filesystem uses an external journal and that journal is (very explicitly) not copied along, the destination must be mounted without journal (or, as I have done, the journal needs to be removed). This is, of course, completely okay for a backup filesystem.
tune2fs -f -O ^has_journal $filesystem