Page MenuHomePhabricator

Service mesh envoy does not treat incoming connections as local
Open, MediumPublic

Description

For T353460 we tried to explicitly disable a configured retry (at the mw-api-int-async-ro listener) by sending x-envoy-max-retries: 0. Unfortunately this does not have any effect as envoy does not consider the connection to be local/trusted unless x-forwarded-for is set to only one RFC1918 or RFC4193 address.

This behavior can be changed by setting use_remote_address: true:

If set to true, the connection manager will use the real remote address of the client connection when determining internal versus external origin and manipulating various headers. If set to false or absent, the connection manager will use the x-forwarded-for HTTP header.

I would think that we can safely set use_remote_address: true for all mesh listeners that we generate automatically but there might ofc. be implications that I'm missing.
In this context it might also make sense to change the listen address for all generated listeners to 127.0.0.1 instead of 0.0.0.0 to avoid making them accessible by accident (currently that would need an additional networkpolicy).

Event Timeline

Scott_French added subscribers: RLazarus, Scott_French.

From a quick spot check in puppet and deployment-charts, it doesn't look like we've subsequently done this widely, though there are limited use cases in which use_remote_address: true is set.

It still sounds like this makes sense to do, though I'm not quite sure whether some of the recent changes around internal_address_config for compatibility with newer envoy versions might change this story.

@JMeybohm or @RLazarus - If either of you happen to know off hand, that would be appreciated. Thanks!

Not offhand, but your reading sounds believable to me. I'd also be interested in whether the change to X-Forwarded-For (Envoy wouldn't append the remote address to it anymore) would cause any problems for anyone depending on that, but I wouldn't imagine so.