Page MenuHomePhabricator

orchestrator: Add service monitoring
Open, LowPublic

Description

We need monitoring+alerting for orchestrator's health.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone

Event Timeline

Kormat created this task.Oct 23 2020, 1:42 PM

Change 636067 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] orchestrator: add monitoring for process and TCP port

https://gerrit.wikimedia.org/r/636067

Marostegui triaged this task as Low priority.Oct 26 2020, 6:35 AM
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui added a subscriber: Marostegui.

We don't have it in production, so putting this to low as we aren't on a hurry for this as of today

Change 636067 merged by Dzahn:
[operations/puppet@production] orchestrator: add monitoring for process and TCP port

https://gerrit.wikimedia.org/r/636067

Dzahn added a subscriber: Dzahn.Oct 26 2020, 5:39 PM

New checks have been added to Icinga:

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=orchestrator

But notifications for everything on this new host are disabled. Up to you when you want to enable them.

Dzahn moved this task from Blocked to Ready on the DBA board.Oct 26 2020, 5:40 PM

Thank you Daniel!
This looks good for now, so far we are going to keep notifications disabled on the host as we are doing many changes still, some of which involves restarts, breakages etc!
Thanks for helping out!