As part of T321736, we created a virtual machine on the analytics namesapce in https://horizon.wikimedia.org/ that runs a custom gitlab runner which allows the running script to launch more containers.

(This is colloquially referred as 'docker on docker' but we specifically not do that, we do 'docker besides docker' as it doesn't require privileged mode and can be more performant. [see here for more info]).

Since we run this virtual machine on Horizon (aka CloudVPS), we were able to puppetize the installation of docker. But we definitely took some liberties and manual steps for the rest of the configuration, including manually adding more available storage and manually installing the gitlab runner package. All these manual steps are documented on T321736.

In this task, we should:

• Revisit the creation of such runner instances and puppetize it properly.
• Likely shutdown and replace the current gitlab-docker-runner.analytics.eqiad1.wikimedia.cloud instance.
• Test that everything works as expected.
• Make such puppet gitlab runner templates available for other folks at Wikimedia.