+1 to this task.

The rationale it the same as the parent tasks and is not new: anything that crosses the VPS instance -> production barrier is unsafe and undesired. Imagine e.g. a Graphite or statsd vulnerability; especially given that these are not products that were designed to run over unsafe networks or the Internet and haven't been scrutinized as much.

That said, I wouldn't say that the requirement is to move to a VPS per se. If, for instance, at some point in the future bare metal servers in the labs realm become possible, that could be a possibility too. The ask (at least my ask!) would be for this instance-supporting service to live outside of the production realm (and 10/8 space etc.), and I don't have strong opinions on where else they should live :)

In T207543#4684881, @aborrero wrote:

We have dedicated hardware for this, in the case of labmon1002, a fairly new server which was put into works some months ago.

Yeah, can't just waste hardware, but at the same time I'm not sure this shouldn't be done purely on that basis. Is that hardware particularly special? Maybe there is another use for it?

In T207543#4684881, @aborrero wrote:

I'm unaware of any debate to move away from the current model.

That's what this task is for :)

+1 to this, I'd include prometheus as a service we should set up for labs projects that want to use it too. T211640: Grafana, icinga, prometheus in cloud-analytics project