When our infrastructure is being blocked by 429s (Too Many Requests), I want to be able to see alarm being triggered on PagerDuty, so that I can be a able to timely react to the such event.
problem: When we are being blocked by 429 requests we wait until we are allowed to query the API again.
Acceptance criteria
When we are blocked by 429s support engineer gets a notification.
ToDo
- add an alarm to IaC
- trigger PagerDuty alarm based on that
- test it
Notes
We can monitor the logs of structured-data service to do this, all of the logs that we need to catch will contain 429 Too Many Requests within the log entry. Here's an example:
/app/submodules/wmf/wmf.go:589: 429 Too Many Requests:
{
"error": {
"code": "too_many_requests",
"message": "AlimitednumberofparallelconnectionsperIPisallowed."
}