Page MenuHomePhabricator

restbase/cassandra: skip (some) icinga monitoring if on "dev" machines
Closed, ResolvedPublic

Description

don't have (the restbase specific) Icinga monitoring checks on "restbase-dev" machines, only on the real prod machines

Event Timeline

Dzahn created this task.Mar 6 2018, 9:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2018, 9:47 PM

Change 419255 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] restbase: allow to skip monitoring, disable on dev hosts

https://gerrit.wikimedia.org/r/419255

Change 419255 merged by Dzahn:
[operations/puppet@production] restbase: allow to skip monitoring, disable on dev hosts

https://gerrit.wikimedia.org/r/419255

Change 419337 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] restbase: skip root URL monitoring on dev cluster, pt2

https://gerrit.wikimedia.org/r/419337

Change 419337 merged by Dzahn:
[operations/puppet@production] restbase: skip root URL monitoring on dev cluster, pt2

https://gerrit.wikimedia.org/r/419337

Dzahn updated the task description. (Show Details)Mar 14 2018, 1:29 AM

on einsteinium after running puppet on restbase-dev1006:

 define service {
-# --PUPPET_NAME-- restbase-dev1006 restbase_http_root
-	active_checks_enabled          1
-	check_command                  check_http_port_url!7231!/
...

The "root URL" check that kept alerting on dev machines is now removed.

Change 419339 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cassandra/icinga: make monitoring configurable, skip on dev

https://gerrit.wikimedia.org/r/419339

Dzahn triaged this task as Normal priority.Mar 15 2018, 8:51 PM

partially resolved

Dzahn moved this task from Backlog to In progress on the observability board.Mar 19 2018, 3:08 PM
Dzahn added a comment.EditedMar 19 2018, 3:36 PM

partially resolved because https://gerrit.wikimedia.org/r/419339 is done but additionally i still would like this one merged:

https://gerrit.wikimedia.org/r/#/c/419084/

which allows us to skip systemd monitoring in base, generally

and then skip it for restbase-dev

I merged that and it works (thanks for review).

Then i used it to skip systemd monitoring on gerrit2001 (because of T176532)

Meanwhile the systemd status on restbase-dev is already OK again by itself though.

So i'll call this resolved because now we have the option to do that and we can do it if we need it.

Dzahn renamed this task from restbase: skip icinga monitoring if on "dev" machines to restbase: skip (some) icinga monitoring if on "dev" machines.Mar 22 2018, 10:36 PM
Dzahn closed this task as Resolved.

Change 419339 merged by Dzahn:
[operations/puppet@production] cassandra/icinga: make monitoring configurable, skip on dev

https://gerrit.wikimedia.org/r/419339

Dzahn added a comment.Apr 11 2018, 5:32 PM

The change above will now ensure that cassandra Icinga checks are not added if on the dev cluster. We don't see the results yet because puppet is currently disabled on restbase-dev due to ongoing upgrades.

Dzahn renamed this task from restbase: skip (some) icinga monitoring if on "dev" machines to restbase/cassandra: skip (some) icinga monitoring if on "dev" machines.Apr 11 2018, 5:32 PM
Dzahn added a subscriber: Eevans.