Last week, we completed a piece of long-neglected work relating to Puppet, the tool that manages the configuration of every virtual machine in our cloud. Historically, each VM has received its configuration from a physical, production server (the 'puppetmaster'). This meant that there was a constant chatter of traffic back and forth between each VM and unrelated networks and hardware sitting in Wikimedia production. Now, the puppetmasters are located on VMs, so all of that chatter is internal to Cloud Services.
Generally, we like to think of the cloud as an isolated sandbox, a safe place for volunteering and experimentation. Any tight links between the cloud and production require extra vigilance; as we sever those links we can worry a bit less about issues (security and otherwise) bleeding back and forth between the cloud and the public wikis.
A notable thing about this move is that nearly all the work was done by a technical volunteer, @Krenair. Krenair updated the code that runs the in-cloud puppetmasters, built out the server cluster, and designed the migration flow that transferred control over from the old controllers. It was his hard work (done on top of his unrelated day job) that moved this task from long-neglected to a box with a check mark.
Quite a few Cloud Services projects are partially (or, in some cases, completely) maintained and managed by technical volunteers. Not only does this allow us to run infrastructure well beyond the capacity of our small team, it's also a clear success in the mission of the Technical Engagement team (of which Cloud Services is a part). We work to build technical capacity in the community, and when volunteers start doing our job for us, we know we've succeeded. Almost all levels of access are available to trusted volunteers, and getting permission to hack on the WMCS infrastructure is not as hard as you might think. Come and join us!