This is a followup/breakout from T350360: Evaluate "drop in" replacement for nrpe scripts specifically for checks/alerts that require more information and/or have larger outputs. In general all alerts for which there are followup actions, either informative only (gather more output to help operators, open tasks) and/or possibly towards auto-remediation. The original use case in the task is to replace raid checks which make use of nrpe and icinga's event handler functionality to call back the host for raid status and finally open a task.

I'll quote @Volans here for more context, though feel free to edit the description as needed

In T350360#10488295, @Volans wrote:

As for the "large output" kind of script problem one possible option could be to pair it with a project we've talked about in I/F but never get to do it: the possibility to trigger some cookbooks via some HTTP api/UI both for scheduling and auto-remediation capabilities. This could be a third use case, the reporting triggered by some alerts.

How this could look like at high level:

• Convert the existing scripts to just emit one metric with the status and no output. For example we could use the nagios check exit code as value (0, 1, 2, 3)
• Have alerts setup based on those metrics that trigger a webhook
• Have an internal HTTP endpoint where the webhook can authenticate and trigger a run of a cookbook that could either re-run the check script on the host with some parameter to get the full output, or have the script save the status on a file and have the cookbook retrieve it, etc... At that point the cookbook could use the output anywhere else (phabricator task or paste, email, etc...)

There are a bunch of implementation specs to be decided on the cookbook API side (auth, execution, concurrency management, failures management), but is something that we'd need to investigate and decide anyway if we go ahead with the project also for the other use cases.
If this is something you're interested in we can surely find some time to discuss it between observability and infrastructure foundations also for planning purposes.

Solution outline

This is the summary of a meeting with @elukey @tappof @Volans and @fgiunchedi about how to solve the spicerack/alertmanager integration.
There are two main parts: a webservice to act as an alertmanager webhook receiver and the cookbook(s) themselves. Note that the solution is focused on raid failure handling, however it is generic and can be later expanded to more cookbooks to perform more autoremediation actions.

Webservice

• Receives and parses alerts from alertmanager
• Has a list/mapping of allowed alerts to their respective cookbook(s)
• calls the (yet to be named) cookbook for raid failure handling, passing arguments such as hostname and raid type

Cookbook

• Is designed to run unattended and be called synchronously by the webservice
• SSH to the host and calls the existing raid handler scripts based on raid type, receives exit status + output
• Creates a task with appropriate dcops tags based on the host' site
• Optionally silences the received alert

Alertmanager

• Receives raid failure alerts (one per raid type) and calls the webservice with said alert

Open issues/questions

• wmflib currently lacks the ability to open tasks, which will need to be added
• The simplest solution is for the webservice to subprocess into spicerack; we need to understand the request handling concurrency model of the webservice. In other words make sure that multiple independent requests each can spawn a call to spicerack without blocking others.
• There will be a need to perform privilege escalation webservice -> spicerack, possibly with a restricted sudo spicerack like we do for dcops
• We need to point operators to cookbook logs, perhaps print information in the task itself on where to find diagnostic information
• Understand what the semantic is to handle failures, on HTTP errors for the webhook alertmanager will retry the notification until success