Page MenuHomePhabricator

blackbox alerts: add instance names to tickets created by alerting
Open, Needs TriagePublic

Description

status quo:

We are using prometheus::blackbox::check::http in puppet to create monitoring checks.

In alertmanager.yaml we are using webhooks_config to automatically create tickets in Phabricator for alerts and this works well for us.

An example would be T355995.

In the webhooks config we only pass a dashboard URL and Phabricator tag.

feature request:

Add the host name and port (aka "instance") of the relevant service to the ticket title.

Currently all such tickets are just called "ProbeDown" upon creation.

Then we manually rename them to be more descriptive about what exactly is down.

But manual renames are work and lead to inconsistent ticket names that are hard to sort.

The tickets already have the instance: value in the body. For example instance: vrts1001:1443 in the ticket linked above.

Can this be added to the task title?

Related Objects

Event Timeline

Dzahn renamed this task from blackbox alerts: add host names to tickets created by alerting to blackbox alerts: add instance names to tickets created by alerting.Jan 31 2024, 7:30 PM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)

Yes this is possible, albeit a bit clunky at the moment. The way we do it for e.g. dcops tasks is group the alerts on instance and then change the title for the webhook (url-encoded) to create tasks with instance in the title. This also means you'll be getting a ticket per-host, let me know if you'd like assistance with this