Currently there are a few alerts defined in Grafana (as distinct from those defined in the alerts repo). Some of these alerts just don't work (and should probably be cleaned up), but some of these appear to be in use.

We should decide if this is going to be a pattern we officially support in future, and if so we should write an official policy on what level of support Observability will be providing for it. Additionally we should provide instructions on how to write effective alerts, how they will integrate with alertmanager and so on. We should also clarify (and either configure or disable) things like email alerting, which doesn't work and probably never has despite some alerts using it. We should also be explicit about which alerts should not be in Grafana and should be in the alerts repo instead - higher importance, more critical or more general infrastructure alerts for example.

Considerations:

• Given the significant impact of a test or poorly defined alert firing, we should either remove or heavily restrict the ability of user defined alerts to page (unless this is a requirement of currently defined alerts)
• We deliberately don't use fine-grained permissions for authenticated users in grafana, so limiting access here is not really in scope. Are there things we need to consider in this context?
• Alerts and alert contacts are currently public. This isn't a major concern currently as the alerts repo is also public, but it should be impressed upon users as custom alert contacts have the potential to be used differently to those defined in the alerts repo.