On 2015-02-04, twice a Hadoop name node died with "Java heap space".
On 2015-02-07, both Hadoop name nodes died with "Java heap space" errors around the same time, so fail-over was not possible.
Upon manually restarting the name nodes, they died again with "Java heap space" before they could properly start.
Hence, HDFS was down and could not be brought up.

Manually bumping the heap size for the name nodes from the default of 1GB to 2GB worked, and allowed to restart the name nodes.

Let's puppetize a 2GB heap for the name nodes.