Installing the new cp hosts in eqiad (T349244) we noticed a strange behavior while requesting resources from the new text hosts (but same behavior applies when the request is made also from other hosts):
```
fabfur@cp1100:~$ curl -H "X-Forwarded-For: 2620:0:861:ed1a::1, 10.64.16.240" -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache'
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private, must-revalidate, max-age=0
```
This, being uncacheable by ATS, results in a `pass` to Varnish.
Some notes to help troubleshooting:
* The XFF header is populated by both HAProxy and Varnish
* This happens only in eqiad|codfw, when the XFF is set to the local text-lb address:
```
vgutierrez@cp6009:~$ for dc in eqiad codfw esams ulsfo eqsin drmrs; do echo $dc && curl -H "X-Forwarded-For: $(dig +short text-lb.$dc.wikimedia.org), 10.136.0.10" -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.org" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 |egrep -i "Expires|cache"; done
eqiad
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private, must-revalidate, max-age=0
codfw
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private, must-revalidate, max-age=0
esams
< Cache-Control: s-maxage=86400, must-revalidate, max-age=0
ulsfo
< Cache-Control: s-maxage=86400, must-revalidate, max-age=0
eqsin
< Cache-Control: s-maxage=86400, must-revalidate, max-age=0
drmrs
< Cache-Control: s-maxage=86400, must-revalidate, max-age=0
```
* This happens only on `en.wikipedia.org` domain, as a counter-example:
```
vgutierrez@cp6009:~$ for dc in eqiad codfw esams ulsfo eqsin drmrs; do echo $dc && curl -H "X-Forwarded-For: $(dig +short text-lb.$dc.wikimedia.org), 10.136.0.10" -H "X-Forwarded-Proto: https" -H "Host: it.wikipedia.org" https://appservers-rw.discovery.wmnet/wiki/Friedrich_von_Kenner -v -o /dev/null -s 2>&1 |egrep -i "Expires|cache"; done
eqiad
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
codfw
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
esams
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
ulsfo
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
eqsin
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
drmrs
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
```
* Not setting XFF header or setting it only with the **private** address results in a cacheable response:
```
fabfur@cp1100:~$ curl -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache'
< Cache-Control: s-maxage=86400, must-revalidate, max-age=0
fabfur@cp1100:~$ curl -H "X-Forwarded-For: 10.136.0.10" -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache'
< Cache-Control: s-maxage=86400, must-revalidate, max-age=0
fabfur@cp1100:~$ curl -H "X-Forwarded-For: 208.80.154.224" -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache' # 208.80.154.224 is text-lb.eqiad.wikimedia.org
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: private, must-revalidate, max-age=0
```