Description
Details
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| Revert "eventgate-analytics: bump replicas from 20 to 30" | Clément Goubert | operations/deployment-charts | master | +2 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Clement_Goubert | T324994 Incident: 2022-12-09 api appserver worker starvation | |||
| Open | None | T325068 Uneven CPU throttling of eventgate-analytics under load |
Event Timeline
Change 867597 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] Revert "eventgate-analytics: bump replicas from 20 to 30"
Since this incident was caused by a temporary raise in logging volume, and our response was to scale up the bottleneck, this task will only track reverting to the state before the incident.
reverting to the state before the incident
Hm, do we need to revert? I don't mind either way, but +10 replicas (in each DC), probably is okay and will allow us to handle more load if something like this happens again?
I thought we had talked about reverting when the train had passed reverting the root cause, but checking back in my logs, I misremembered.
We have the resources to keep it at 30 replicas at the moment in my opinion. @JMeybohm what do you think?
Hi, The flood of logs is still incoming, the revert of logspam has not been deployed yet so I advise against reducing pods now.
Yes, the commit message of the above changelog makes it very clear it is not to be deployed before the train has ran on Thursday. We might end up not scaling back anyways.
Change 867597 abandoned by Clément Goubert:
[operations/deployment-charts@master] Revert "eventgate-analytics: bump replicas from 20 to 30"
Reason:
Sorry, totally missed that. Yes, resource wise we are absolutely fine keeping the 30 replicas.
No worries, I took a look at the resources and it seemed fine to leave it like that. We can always revisit when/if contention happens with other services.
Removing the sustainability tag as it doesn't seem like there is any related actionable here. @Clement_Goubert if you think that's unfair please undo my change.