Most of the Icinga checks in the parent task are gone, one class of checks that's left is the DNS ones. Those are possible right now because alert hosts have public IPs and thus can reach cloudservices hosts directly.

Also of note that the same dns checks are covered by blackbox exporter on metricsinfra project, which of course works as expected because VMs can reach cloudservices on their public IPs. In the DNS case checks from within the cloud network might be fragile, in the sense that alerts might not even go out if DNS is busted.

For monitoring cloud realm from production realm, moving such checks to blackbox exporter on Prometheus hosts would not work because the hosts are on the prod internal network.

To enable DNS checks and other kinds I think we should establish a blackbox-exporter within cloud realm, that in turn can be queried (proxied) from prometheus hosts like any other blackbox. We will enable TLS auth and possibly source network restrictions on the blackbox so it is available only to authorized hosts.

My understanding is that cloudservices hosts would be a good fit for said vantage point (?) they can obviously reach dns, perhaps other use cases we might have in the future?

Another option is to instead run blackbox-exporter on alert hosts, though that's a different scope in the sense that we'll need to consult with Observability, and there would be only one per site.

Yet another option that doesn't require an http proxy but blends the boundary between realms: run blackbox-exporter on hosts on both realms, the exporter listens on the prod IP and thus prometheus can reach it, and blackbox can talk to the cloud realm too