@Andrew reports that:
One issue that's currently thwarting a lot of our testing is the fact that new VMs in the labtest (and, presumably, labtestn) cluster don't come up properly. Ideally I would be able to create a new VM on labtesthorizon.wikimedia.org and then log into it via a proxycommand in order to verify that all the parts are talking to each other. In reality, that never works and I just wind up scouring the VM boot logs for signs of trouble instead.
This problem consists of many different pieces which I probably won't be able to rattle off coherently, but you can at least see the current problem by creating a VM on labtesthorizon for yourself and then looking at its unhappy log on the horizon interface. They get pretty far but generally break during the puppet handshake.
If and when you feel like building a new image to address whatever issues are happening (if it even is image-specific), instructions for building an image are here:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance#Building_a_Debian_image
And, you should feel free to stop puppet and hotfix things in labtest while exploring.