During the incident review for this page, a few interesting things came up as possible action plans that stemmed from the question "why does a high number of emails in the queue result in a page and what exactly would happen if the queue was left in that state?":
- Should we be looking at all message statuses instead? Including frozen emails (frozen means that exim gave up on delivering the email altogether and ideally those should be cleaned up but it was pointed out that this might not be happening.
- Can we count differently? Do we need to take action immediately? Maybe split alerts based on queue states?
- We could also explore alerting based on the rate of the queue increasing in size.
- Big queue can mean widespread problems e.g. delivering to Google. Splitting mx functions will also help (e.g. inbound vs outbound mail flows)
More details on the incident are in the document linked below.
https://docs.google.com/document/d/1JwxW_31IbhsW-oCOeSddf1Y4OtHTWT1BWawg3mcoLa8/edit#