I noticed from Icinga2 alerts the following:
<icinga2-wm> PROBLEM - Host ORES-web01.Experimental is DOWN: check_ping: Invalid hostname/address - ores-web-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4]with the same message for web-02.
Are those instances no longer existant/in use, and if so is it safe to remove them from Icinga2?
https://tools.wmflabs.org/openstack-browser/project/ores does not list them, but does list ores-web0[456], which suggests they were deleted.
I realize that, but I created this so I can verify that they arent going to be recreated and if not do they determine them to safe to remove from Icinga2
Yes. I killed ores-web-01,2,3 in favor of ores-web-04,5,6. Sorry for the lack of notice! I forgot about icinga checks because we're dealing with an instability issue so icinga hasn't shut up in weeks :| (Note for any passer-by, this is not production. Production is rock solid.)
Change 536775 had a related patch set uploaded (by Zppix; owner: Zppix):
[labs/icinga2@master] Remove ORES-web[01,02].experimental from Icinga2
Change 536775 merged by Zppix:
[labs/icinga2@master] Remove ORES-web[01,02].experimental from Icinga2
Mentioned in SAL (#wikimedia-cloud) [2019-09-15T17:13:36Z] <Zppix> cd /etc/puppet/modules/icinga2; git pull on puppet-paladox. puppet agent -tv on gerrit-mysql to deploy [[gerrit:536775]] T232921
Change 537202 had a related patch set uploaded (by Zppix; owner: Zppix):
[labs/icinga2@master] Add ores-web-[04,05,06].wmflabs to monitoring
Change 537202 merged by Zppix:
[labs/icinga2@master] Add ores-web-[04,05,06].wmflabs to monitoring
Mentioned in SAL (#wikimedia-cloud) [2019-09-16T21:19:53Z] <Zppix> deploy [[gerrit:537202]] for T232921 on gerrit-mysql