Page MenuHomePhabricator

Investigate shorter-lived persistent connections for change-propagation
Open, MediumPublic

Description

In the 2022-03-27 API incident, we found that changeprop kept connections to the API servers open for too long.

Normally those long-lived connections are good for performance -- they keep us from losing time on handshakes -- but in this case, when the load on the API servers became uneven, the persistent connections caused it to stay uneven, until we restarted Envoy on the api servers. If we set a shorter max duration, still reusing the connection but not for so long, that situation will recover on its own.

As far as I can tell this isn't currently supported in changeprop, but I expect the libraries it uses to be able to do so.

Event Timeline

herron triaged this task as Medium priority.Mar 28 2022, 4:18 PM
akosiaris subscribed.

Removing SRE, has already been triaged to a more specific SRE subteam

I'm not sure this is really actionable without any number attached.

We already set a maximum number of client connections we accept in envoy, and I strongly suspect the problem here is actually that we don't use envoy in changeprop, but its own connection pool to connect upstream.

So what we'd need is to support ending persistent connections in changeprop, which would need work from platform more than from SRE.

I'll update the task description accordingly.

Joe renamed this task from Investigate shorter-lived persistent connections for Envoy to Investigate shorter-lived persistent connections for change-propagation.Mar 21 2023, 8:07 AM
Joe edited projects, added: Platform Engineering; removed: envoy, serviceops-deprecated.
Joe updated the task description. (Show Details)