After doing maintenance on 7/25/2019, @JHedden noticed that the mounts were mixed up on cloudstore1009 because the bind mounts weren't re-created on boot. We had to manually run nfs-manage-binds, restart NFS and then let nfs-exportd re-export everything to recover after a reboot.
We have a few ideas here:
- Don't let NFS start on reboot. Keep it manual like the other cluster (labstore1004/5). This seems unnecessary.
- Make nfs-manage-binds into a timer or service that will run at boot or even at intervals (on this cluster only or only on the "active" node of labstore1004/5)
- Simply place the mounts in fstab for this cluster since there are only two of them, and there's no active-passive DRBD set to worry about right now.
We should think about how to proceed and make sure this doesn't surprise us in the future.