The new shelf has been added, and configured. Actual expansion is pending on thin volumes, which itself requires a backport of a recent version of lvm2 (which is nearly complete) - Precise has no working thin volume support.

(Upgrading labstore1001 [to Jessie] has been considered, but given the long downtime and the recent outages we decided against hitting the users again so soon)

Hardware happy in place and visible to the OS.

After another discussion with @faidon that concluded:
<Coren> paravoid: So you think it's better to Jessie up 1002 and switch to that instead?
<paravoid> I think so, yes

That plan involves a (brief) downtime for the switchover, less than 10 minutes, but is futureproof.

Dumb question: IIRC there are two disk arrays each connected to two NFS servers? Switching between the NFS servers requires clients to remount everything (aka reboot)?

Change 194537 had a related patch set uploaded (by coren):
labstore1002 to Jessie

https://gerrit.wikimedia.org/r/194537

@scfc: No, the NFS fsids are the same and the actual service IP is floating, so no remount is required from the clients.

The new filesystem is active and in place; the rsync is in progress (currently at iowait -c Idle) but will take some time. I'll discuss giving it more bandwidth during the ops meeting.

So here is the current picture:

• The new filesystem on thin volumes is in place and contains a copy of the live filesystem, but rsync is unable to keep up with the rate of change so actual downtime is unavoidable to do the actual switchover (a dry run at ionice takes ~20 hours!)
• The new filesystem properly does snapshotting; we have local backups available
• Replication of the latest snapshot to codfw works, but has performance issues until proper exclusions are put in place
• All of those processes are tested and fully working over the new filesystem (which is not the live one)

What's needed for this to be done:

• Upgrade labstore2001 to Jessie before we flip the switch (our last chance to do so)
• Schedule a downtime to do the final rsync between old and new filesystem
  • Same as was planned in January, 24h during which the filesystem will be readonly
  • Once the copy is done, swap out the mountpoints
• Flip the switch:
  • Puppet class to install the scripts and stuff them in crontabs

What we might want to do afterwards:

• Make the snapshots accessible to the endusers? Not trivial, but right now getting files off snapshots requires admin intervention. Sufficient for now, but won't scale.

Status update:

• labstore2001 upgraded
• copy done, mountpoint swap scheduled for today (Mar 30) 22h UTC

Todo:

• Finish review/tweaks of replication code and flip the switch

Is there documentation (with a procedure to follow) for the "cold spare" redundancy yet?

How much additional space (storage expansion) has been made available by this?

@coren: see questions above, thanks!

In T85604#1167208, @mark wrote:

How much additional space (storage expansion) has been made available by this?

An extra 25%, approximately 18T of usable space. In addition, the cleanup required by the transition between filesystems managed to clean another 5-6T of redundant backups leftovers from pmtpa.