Last night (Apr 2), one of the shelves attached to labstore1001 has been slipping in and out of reachability until a complete powercycle. It's been working right since but as those same symptoms appeared the day before and resolved themselves I have no way too know for certain whether that was a coincidence or not.
I've no way to identify the shelf with certainty from the server, but I can make leds blink to show you once you get there. :-)
Due to labstore1002 not having been power cycled, it still sees the cuplrit shelf as lame so it was identified to be the most recent addition to the array. Visual inspection shows no obvious wiring issues and it appears to be working now, fiddling with it may be unwise.
Reducing priority so that it can be examined more closely at the next opportunity (there is a planned switchover test some weeks from now that may be a good opportunity).
The wiring is stable. Verified all the cables were snug in their ports. Resolving this task.