Page MenuHomePhabricator

Log the real X-Client-IP in apache mediawiki logs
Open, LowPublic

Description

It seems that when requests go from LVS -> k8s -> mediawiki, do not have the X-Client-IP header set, which we normally set in varnish. The result is that we find in our apache logs

2021-03-01T17:03:25 47280 10.64.0.71 proxy:unix:/run/php/fpm-www.sock|fcgi://localhost/200 80 POST http://zh.wikipedia.org/w/api.php - application/json - - Mobileapps/WMF zh-cn - - - 10.64.0.71 dbbd739a-7e13-4726-a62f-8644cd53f2f8 -

where we have X-Client-IP: - and X-Forwarded-For: -, and 10.64.0.71 is the mediawiki's server own IP address. To fix this, we will need to have envoy running in out pods to set this header before sending a request to mediawiki.

Event Timeline

akosiaris renamed this task from Log the real X-Client-IP to Log the real X-Client-IP in apache mediawiki logs.Mar 5 2020, 8:55 AM
akosiaris triaged this task as High priority.
akosiaris updated the task description. (Show Details)
akosiaris added subscribers: akosiaris, Joe.

After discussing with @akosiaris, we decided that when a request is made from k8s towards the API, it makes sense for apache to see the pod's IP address in the XFF header. What we want is to have mw* envoy TLS terminator set XFF to a pod's IP address, when neither XFF nor X-Client-IP address is already set. I briefly tried use_remote_address: true in envoy.yaml on an api server:

2021-03-05T17:34:42	87416	10.64.0.71	proxy:unix:/run/php/fpm-www.sock|fcgi://localhost/200	87	POST	http://www.wikidata.org/w/api.php	-	application/json	-	10.192.65.89	Mobileapps/WMF	zh-tw	-	-	-	10.64.0.71	a5df14e4-1cf1-4bad-9251-70c985fb3658	-

where 10.192.65.89 is kubernetes-pod-10-192-65-89.codfw.wmnet.

akosiaris lowered the priority of this task from High to Low.Mar 8 2021, 2:32 PM

After discussing with @akosiaris, we decided that when a request is made from k8s towards the API, it makes sense for apache to see the pod's IP address in the XFF header. What we want is to have mw* envoy TLS terminator set XFF to a pod's IP address, when neither XFF nor X-Client-IP address is already set. I briefly tried use_remote_address: true in envoy.yaml on an api server:

2021-03-05T17:34:42	87416	10.64.0.71	proxy:unix:/run/php/fpm-www.sock|fcgi://localhost/200	87	POST	http://www.wikidata.org/w/api.php	-	application/json	-	10.192.65.89	Mobileapps/WMF	zh-tw	-	-	-	10.64.0.71	a5df14e4-1cf1-4bad-9251-70c985fb3658	-

where 10.192.65.89 is kubernetes-pod-10-192-65-89.codfw.wmnet.

That does more or less what we want, I think. X-Forwarded-For is populated, X-Client-IP is not (as this is only set by the edge caches). X-Envoy-External-Address will also be populated per docs[1] and have the IP of the pod.

We should double check what happens with X-Forwarded-For for requests coming via the edge caches just to verify our understanding of the situation. I expect external IP, cpXXXX IP, cpYYYY IP, local mwIP, with a variation including restbase after the edge caches.

[1] https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-for