ATM the prometheus::blackbox::check::http puppet abstraction requires ip v4/v6 of its target, and defaults to the host/address the check is defined on.
This is normally fine since we want to run probes against the host itself, the abstraction kinda leaks when we'd like to run checks against frontend endpoints like commons.wikimedia.org. In those cases we're passing the either the service IP (in the phabricator case we have that in puppet) or resolving the name at puppet run time, which as @Vgutierrez pointed out has a problem when we're depooling sites.
Ideally I think we should be going for optional ip address and support resolution when the probe runs. The ip v4/v6 are labels of the probe metrics at the moment, so in that case we wouldn't/couldn't have the addresses in the metrics, which might be ok too. cc @jbond