As we've seen, RAID6 has serious performance issues under write heavy load, and isn't especially great at recovering from unclean shutdown (but we could address the latter to some degree with write-intent bitmaps and kernel updates) or disk failure->recovery scenarios.

RAID6 is what we're using in this case, but the same basic arguments apply to all of RAID[456], which are all variations on the same theme of parity. Parity recalculation is expensive in terms of physical disk traffic and CPU load, and tends not to scale well. The reason everyone uses RAID6 instead of RAID5 these days mostly boils down to the math, and that same math also indicates that even RAID6 is slowly becoming unacceptable. Note this ACM paper: http://queue.acm.org/detail.cfm?id=1670144 (quicker/easier zdnet summary here: http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/ ) . It was written in 2009, and we're now 6/10 years through the remaining 10 year life of RAID6 practicality the author mentions there before we have to move away from RAID6 to something based on triple-parity, because the RAID6 math on capacity vs MTBF vs rebuild time, etc keeps getting worse over time.

Regardless of the above there's always price:perf:space tradeoffs between RAID6 and RAID10, but there's also tradeoffs involve in giving any fixed amount of space to the users in the first place. It would be nice if we had twice the available capacity we do today. If there were a rational reason that we didn't, it would be "why waste the money? Nobody's demanded it that hard yet". But if you did it, users would find ways to expand into it. People tend to size their demands to availability in a lot of cases like these (because in truth, their hard need for space is way lower than their apparent usage of it), and I think the tradeoff is virtually always there at today's storage prices to avoid RAID6 and all of the associated tradeoffs, all of which are negative except for the "more space" bit.

All of the above might have different answers if we were using hardware-based enterprise-grade external RAID arrays with dedicated high-performance controllers and caches. But we're not (and I don't think I'd recommend that, either!). We're using Linux software RAID.

Even in the RAID10 world, there are similar looming issues for mirroring based on disk capacity + failure rate math. The short answers there are: write intent bitmaps help in that case (as they do in every such case) to speed rebuild times, and that using a larger count of smaller disks rather than a smaller count of larger disks still helps with those tradeoffs without significant caveat in the RAID10 world as well. You can also start putting hot spare drives in place (to reduce the total loss-of-redundancy window by taking human reaction + physical stuff out of the window), and then later on in the global timeline of these factors, you really have to start considering triple-mirroring for large datasets where periodic backups and hot spares are insufficient protection.

In T96063#1207581, @coren wrote:

Raid 6 is a performance bottleneck but gives us 66% more effective storage than raid10 would in the current configuration. It doesn't mean that moving away from raid6 is inconceivable, but that doing so has a significant cost (as well as being a complicated operation).

Setting priority to Low since this lives in the nice-to-have domain, and that any investigation for phasing NFS out (which is already in the long-term plans) would imply reconsideration of the backing block devices anyways.

With the current stability and performance problems of NFS with RAID6, this is definitely not a "nice to have" but something that needs to be fixed ASAP. Storage capacity is far, FAR less important than stability at this point.

In T96063#1268232, @mark wrote:

With the current stability and performance problems of NFS with RAID6, this is definitely not a "nice to have" but something that needs to be fixed ASAP. Storage capacity is far, FAR less important than stability at this point.

Given that the plans that necessitated the overprovisioning (a seperate Labs in codfw) no longer exists, we do indeed have sufficient space to use raid10 rather than raid6 (60T suffices for primary storage and snapshots, the extra 30T were needed if we had cross-DC storage replicated).

Migrating the logical volumes between raid6 and raid10 will be a several-step manoeuvre as we need to succesively evacuate the individual raid devices (being 1:1 with physical shelves), bebuild the array as raid10, and move the volume from pv subset to pv subset. This can be done while the filesystem is live, but is both a delicate operation and very I/O intensive (on the plus side, the writes to the new pv will not be amplified by the raid6 setup).

Because the actually available number of pe will diminish after each successive conversion, the actual number of operations will likely need to increase (think towers of hanoi).

@coren: Where can I see the mapping of raid array (md125 etc) to shelf? Is this documented?

@mark: It's in the slides (https://commons.wikimedia.org/wiki/File:WMF_Labs_storage_presentation.pdf) but also ridiculously straightforward: shelves are mapped 1:1 to /dev/md/slice[1-5].

A note: while it will probably increase the amount of necessary juggling, the entire setup would be immensely improved with raid10 if - rather than one shelf one array - we put 6/6 drives per array on /consecutive/ shelves (so 1-2, 2-3, 3-4, 4-5, 5-1). One average, this will not generally improve disk bandwidth but it will make the arrays imprevious to loosing a shelf briefly.

Can you work out a plan, and list all the individual steps (ideally with command line invocations) on this ticket?

The plan is to gradually evacuate the pv on individual raid arrays with pvmove, reconfigure the freed raid arrays with raid 10, and recreate pv on the new arrays to be used as destination for further moves. slice5 is currently mostly empty already, and slice1 is currently allocated only to the (now obsolete) safety copy (pre-thin volumes) store/project. So:

1. drop the store/project volume
2. remove slice1 pv
3. pvmove the contents of slice5 away
4. drop slice5 pv
5. create the new 'slice51' array accross the end of slice5/start of slice1
6. pvmove what is left on slice2 away (most was the rest of store/project, the rest fits)
7. drop slice2
8. create 'slice12' (end of slice1, start of slice2)

At this point, we don't have quite enough room for removing another pv with the volume sizes as they are. We can safely shave approximately 12T off the thin pool without risking running out of space - freeing most of slice4

9. pvmove slice4 away (slice3 is the busiest, keeping for last)
10. drop slice4
11. create 'slice45'
12. pvmove slice3 away (this leaves us with roughly 4t to spare)
13. drop slice4
14. create slice23 and slice34
15. restore the thin pool size

A further optimization may be possible at that point by moving around store/journal to live on the least use slice.

An alternative plan, based on input from @mark, that front loads the thin pool move to give performance improvement earlier. With a bit of extra juggling (because it's a bit harder to free the second slice in that scenario):

1. drop the store/project volume
2. remove slice1 pv
3. pvmove the contents of slice5 away to free room on slice2
4. drop slice5 pv
5. create slice51
6. reduce the store/space thin pool to 20T
7. pvmove the tail end of store/space from slice4 to slice51
8. drop slice4 pv
9. create slice45
10. pvmove tail end of slice3 to what is left of slice51
11. pvmove rest of slice3 to slice45
12. drop slice3 pv
13. create slice34
14. pvmove remainder of slice2 to slice34
15. drop slice2 pv
16. create slice12 and slice23
17. restore thin pool size

This seems reasonable yes. Let's move ahead with this after the new backups have finished (codfw + eqiad).

The backup is complete, and the snapshot has been removed - freeing slice5 completely.

Resuming now at step 4.

Problem in step 6:

# lvreduce -L 22T store/space
  Thin pool volumes cannot be reduced in size yet.

Where "yet" means "ever" because CLOSED WONTFIX upstream,

Recalculating the necessary volume moves; this remains doable but may now require intermediate steps and a temporary pv.

Now needs to be:

6. pvmove journal, keys, tmeta and the tail end of space from slice2 to slice51 [1189581 free pe on slice51]
7. remove slice2 pv
8. create slice12 pv [4050187 pe available for 51+12]
9. add dumps pv (from the internal controller) [2383838 extra pe]
10. pvmove first 2383838 extents from slice3 to dumps
11. pvmove remaining 2383839 extents from slice3 to slice12 [476767 free left]
12. remove slice3 pv
13. create slice23 pv
14. pvmove slice4: 1189581->slice51, 476767->slice12, 2860606->slice23 [240723 left]
15. create pv from free "half" of ex-slice5 [1430303 pe]
16. pvmove 240723 remaining of slice4 to that temp pv
17. remove slice4
18. create slice34
19. pvmove temp pv to slice34
20. remove temp pv
21. create slice45
22. pvmove dumps pv to slice34, slice45

Phew.

journal, keys and tmeta migrated to slice51

After some wrangling with block io sizes (thin lvm volumes refuse block transations bigger than the page size), the move of space from slice2 to slice51 is now in progress, at a glacial pace.

(examination of the lvm mirror code used to copy the data shows that it serializes reads and writes with barriers, so the per-block latency is very high)

There is a screen running on labstore1001 with the pvmove, and ongoing status.

In T96063#1372238, @coren wrote:

journal, keys and tmeta migrated to slice51

After some wrangling with block io sizes (thin lvm volumes refuse block transations bigger than the page size), the move of space from slice2 to slice51 is now in progress, at a glacial pace.

(examination of the lvm mirror code used to copy the data shows that it serializes reads and writes with barriers, so the per-block latency is very high)

There is a screen running on labstore1001 with the pvmove, and ongoing status.

I think this was largely because the underlying new RAID10 md device was also doing a resync, at a glacier pace of 1000 KB/s as well. I attempted pausing it, but it automatically restarted. It seems very suboptimal to have both running at once, so I've aborted the pvmove instead, using:

pvmove --abort

And the resync is now going much faster. After that completes, we can restart the pvmove using:

pvmove --verbose /dev/md124 /dev/md122

In T96063#1373448, @mark wrote:

I think this was largely because the underlying new RAID10 md device was also doing a resync, at a glacier pace of 1000 KB/s as well. I attempted pausing it, but it automatically restarted. It seems very suboptimal to have both running at once, so I've aborted the pvmove instead, using:

This did speed up the resync a lot, but it was still going to take 1-2 days. Therefore I've restarted the pvmove, which I think allows the entire migration to finish sooner and be better overall. But before we take any steps after this first pvmove, let's reevaluate if we want to finish for the resync or not. If the next steps don't affect this raid array, they can happen in parallel.

The filesystem crashed caused us to... improvise around this plan a great deal. All but one project has been switched to a restored backup on raid10 - what remains to do now is to clean up the rest of the shelves once all that can be salvaged has been and raid10 them.