Change Details

When our infrastructure is being blocked by 429s (Too Many Requests), I want to be able to see alarm being triggered on PagerDuty, so that I can be a able to timely react to the such event. **Acceptance criteria** When we are blocked by 429s support engineer gets a notification. **ToDo** - [ ] add an alarm to IaC - [ ] trigger PagerDuty alarm based on that - [ ] test it **Notes** We can monitor the logs of `structured-data` service to do this, all of the logs that we need to catch will contain `429 Too Many Requests` within the log entry. Here's an example: ``` /app/submodules/wmf/wmf.go:589: 429 Too Many Requests: { "error": { "code": "too_many_requests", "message": "AlimitednumberofparallelconnectionsperIPisallowed." } ```

When our infrastructure is being blocked by 429s (Too Many Requests), I want to be able to see alarm being triggered on PagerDuty, so that I can be a able to timely react to the such event. **Acceptance criteria** When we are blocked by 429s support engineer gets a notification. **ToDo** - [ ] add an alarm to IaC - [ ] trigger PagerDuty alarm based on that - [ ] test it **Notes** We can monitor the logs of `structured-data` service to do this, all of the logs that we need to catch will contain `429 Too Many Requests` within the log entry. Here's an example: ``` /app/submodules/wmf/wmf.go:589: 429 Too Many Requests: { "error": { "code": "too_many_requests", "message": "AlimitednumberofparallelconnectionsperIPisallowed." } ``` ==Related task https://phabricator.wikimedia.org/T342171

When our infrastructure is being blocked by 429s (Too Many Requests), I want to be able to see alarm being triggered on PagerDuty, so that I can be a able to timely react to the such event. **Acceptance criteria** When we are blocked by 429s support engineer gets a notification. **ToDo** - [ ] add an alarm to IaC - [ ] trigger PagerDuty alarm based on that - [ ] test it **Notes** We can monitor the logs of `structured-data` service to do this, all of the logs that we need to catch will contain `429 Too Many Requests` within the log entry. Here's an example: ``` /app/submodules/wmf/wmf.go:589: 429 Too Many Requests: { "error": { "code": "too_many_requests", "message": "AlimitednumberofparallelconnectionsperIPisallowed." } ``` ==Related task https://phabricator.wikimedia.org/T342171