Page MenuHomePhabricator

Increase wikifeeds capacity to sustain higher peak loads
Closed, ResolvedPublic

Description

From 2021-09-06_Wikifeeds:

  • Increase the capacity for wikifeeds:

We currently have 6 replicas for wikifeeds. During the outage, wikifeeds itself wasn't throttled, but the envoy based tls-proxy was causing some connection failures due to timeouts.

The actual traffic spike was more than 400%, but increasing replicas by so much isn't going to help. As we anyway witnessed, 6 replicas were able to withstand higher loads than usual. A 100% increase is probably going to avoid similar issues at the future and isn't particularly expensive.

Event Timeline

Change 724397 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] wikifeeds: Increase capacity by 100%

https://gerrit.wikimedia.org/r/724397

Change 724397 merged by jenkins-bot:

[operations/deployment-charts@master] wikifeeds: Increase capacity by 100%

https://gerrit.wikimedia.org/r/724397

akosiaris claimed this task.

Merged and deployed.

I 'll resolve this one. It should allow us to not even notice a similar issue as the one that gave birth to this task.