Page MenuHomePhabricator

Have ORES-web[01,02] been removed?
Closed, ResolvedPublic

Description

I noticed from Icinga2 alerts the following:

<icinga2-wm> PROBLEM - Host ORES-web01.Experimental is DOWN: check_ping: Invalid hostname/address - ores-web-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4]

with the same message for web-02.

Are those instances no longer existant/in use, and if so is it safe to remove them from Icinga2?

Event Timeline

Zppix created this task.Sep 14 2019, 3:27 PM
Restricted Application added a project: User-Zppix. · View Herald TranscriptSep 14 2019, 3:27 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Reedy renamed this task from ORES-web[01,02] not existent? to Have ORES-web[01,02] been removed?.Sep 14 2019, 7:37 PM

https://tools.wmflabs.org/openstack-browser/project/ores does not list them, but does list ores-web0[456], which suggests they were deleted.

Zppix added a comment.Sep 14 2019, 7:48 PM

https://tools.wmflabs.org/openstack-browser/project/ores does not list them, but does list ores-web0[456], which suggests they were deleted.

I realize that, but I created this so I can verify that they arent going to be recreated and if not do they determine them to safe to remove from Icinga2

Peachey88 updated the task description. (Show Details)Sep 15 2019, 12:01 AM

Yes. I killed ores-web-01,2,3 in favor of ores-web-04,5,6. Sorry for the lack of notice! I forgot about icinga checks because we're dealing with an instability issue so icinga hasn't shut up in weeks :| (Note for any passer-by, this is not production. Production is rock solid.)

Change 536775 had a related patch set uploaded (by Zppix; owner: Zppix):
[labs/icinga2@master] Remove ORES-web[01,02].experimental from Icinga2

https://gerrit.wikimedia.org/r/536775

Change 536775 merged by Zppix:
[labs/icinga2@master] Remove ORES-web[01,02].experimental from Icinga2

https://gerrit.wikimedia.org/r/536775

Mentioned in SAL (#wikimedia-cloud) [2019-09-15T17:13:36Z] <Zppix> cd /etc/puppet/modules/icinga2; git pull on puppet-paladox. puppet agent -tv on gerrit-mysql to deploy [[gerrit:536775]] T232921

Zppix added a comment.Sep 15 2019, 5:15 PM

@Halfak do you want to us to add ores[04,05,06] to Icinga2?

Yes. Thank oyu.

Change 537202 had a related patch set uploaded (by Zppix; owner: Zppix):
[labs/icinga2@master] Add ores-web-[04,05,06].wmflabs to monitoring

https://gerrit.wikimedia.org/r/537202

Change 537202 merged by Zppix:
[labs/icinga2@master] Add ores-web-[04,05,06].wmflabs to monitoring

https://gerrit.wikimedia.org/r/537202

Mentioned in SAL (#wikimedia-cloud) [2019-09-16T21:19:53Z] <Zppix> deploy [[gerrit:537202]] for T232921 on gerrit-mysql

Zppix closed this task as Resolved.Sep 17 2019, 12:57 AM
Zppix claimed this task.