Page MenuHomePhabricator

Investigate increase and fluctuation in max CPU for linkrecommendation-internal container
Open, MediumPublic

Description

image.png (780×1 px, 174 KB)

Perhaps related to a deployment last week.

image.png (1×2 px, 671 KB)

Event Timeline

It seems to start when this change is deployed:

08:08 urbanecm@deploy1002: Synchronized wmf-config/ProductionServices.php: d149208: Use service-proxy to connect to linkrecommendation (T302719) (duration: 00m 49s)

But I don't see how that would impact the CPU usage in the service.

kostajh triaged this task as Medium priority.Mar 7 2022, 1:35 PM

This change seems to also come with a lower avg latency.
The values from envoy-telemetry-k8s have not changed (although I wonder why that measures way lower latency) fwiw.

What we usually see when services have moved behind the service-proxy is that latency for the caller is decreased (because of less overhead from new connections, tls handshakes,...) and that in some cases means the caller gets more requests out (which does not seem to be the case here