https://gerrit.wikimedia.org/r/327734 and https://wikitech.wikimedia.org/w/index.php?title=Hiera%3ADeployment-prep&type=revision&diff=1168635&oldid=1125290 added some ferm rules which appear on 12 deployment-prep boxes (deployment-imagescaler[01-02].deployment-prep.eqiad.wmflabs,deployment-jobrunner02.deployment-prep.eqiad.wmflabs,deployment-mediawiki[04-07].deployment-prep.eqiad.wmflabs,deployment-mira.deployment-prep.eqiad.wmflabs,deployment-snapshot01.deployment-prep.eqiad.wmflabs,deployment-tin.deployment-prep.eqiad.wmflabs,deployment-tmh01.deployment-prep.eqiad.wmflabs,deployment-videoscaler01.deployment-prep.eqiad.wmflabs):
10_prometheus-apache_exporter:&R_SERVICE(tcp, 9117, (@resolve((deployment-prometheus01.deployment-prep.eqiad.wmflabs)) @resolve((deployment-prometheus01.deployment-prep.eqiad.wmflabs), AAAA))); 10_prometheus-hhvm-exporter:&R_SERVICE(tcp, 9192, (@resolve((deployment-prometheus01.deployment-prep.eqiad.wmflabs)) @resolve((deployment-prometheus01.deployment-prep.eqiad.wmflabs), AAAA)));
Which all seems fine, but puppet showed ferm failing to start. Indeed it fails manually too:
root@deployment-mediawiki06:~# /usr/sbin/ferm /etc/ferm/ferm.conf Error in /etc/ferm/conf.d/10_prometheus-apache_exporter line 4: deployment-prometheus01.deployment-prep.eqiad.wmflabs ) , AAAA ) <-- DNS query for 'deployment-prometheus01.deployment-prep.eqiad.wmflabs' failed: NXDOMAIN
But, that query doesn't return NXDOMAIN:
root@deployment-mediawiki06:/etc/ferm/conf.d# host deployment-prometheus01.deployment-prep.eqiad.wmflabs deployment-prometheus01.deployment-prep.eqiad.wmflabs has address 10.68.20.247