Installing the new cp hosts in eqiad (T349244) we noticed a strange behavior while requesting resources from the new text hosts (but same behavior applies when the request is made also from other hosts):
fabfur@cp1100:~$ curl -H "X-Forwarded-For: 2620:0:861:ed1a::1, 10.64.16.240" -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache' < Expires: Thu, 01 Jan 1970 00:00:00 GMT < Cache-Control: private, must-revalidate, max-age=0
This, being uncacheable by ATS, results in a pass to Varnish.
Some notes to help troubleshooting:
- The XFF header is populated by both HAProxy and Varnish
- This happens only in eqiad|codfw, when the XFF is set to the local text-lb address:
vgutierrez@cp6009:~$ for dc in eqiad codfw esams ulsfo eqsin drmrs; do echo $dc && curl -H "X-Forwarded-For: $(dig +short text-lb.$dc.wikimedia.org), 10.136.0.10" -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.org" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 |egrep -i "Expires|cache"; done eqiad < Expires: Thu, 01 Jan 1970 00:00:00 GMT < Cache-Control: private, must-revalidate, max-age=0 codfw < Expires: Thu, 01 Jan 1970 00:00:00 GMT < Cache-Control: private, must-revalidate, max-age=0 esams < Cache-Control: s-maxage=86400, must-revalidate, max-age=0 ulsfo < Cache-Control: s-maxage=86400, must-revalidate, max-age=0 eqsin < Cache-Control: s-maxage=86400, must-revalidate, max-age=0 drmrs < Cache-Control: s-maxage=86400, must-revalidate, max-age=0
- This happens only on en.wikipedia.org domain, as a counter-example:
vgutierrez@cp6009:~$ for dc in eqiad codfw esams ulsfo eqsin drmrs; do echo $dc && curl -H "X-Forwarded-For: $(dig +short text-lb.$dc.wikimedia.org), 10.136.0.10" -H "X-Forwarded-Proto: https" -H "Host: it.wikipedia.org" https://appservers-rw.discovery.wmnet/wiki/Friedrich_von_Kenner -v -o /dev/null -s 2>&1 |egrep -i "Expires|cache"; done eqiad < Cache-Control: s-maxage=1209600, must-revalidate, max-age=0 codfw < Cache-Control: s-maxage=1209600, must-revalidate, max-age=0 esams < Cache-Control: s-maxage=1209600, must-revalidate, max-age=0 ulsfo < Cache-Control: s-maxage=1209600, must-revalidate, max-age=0 eqsin < Cache-Control: s-maxage=1209600, must-revalidate, max-age=0 drmrs < Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
- Not setting XFF header or setting it only with the non text-lb address results in a cacheable response:
fabfur@cp1100:~$ curl -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache' < Cache-Control: s-maxage=86400, must-revalidate, max-age=0 fabfur@cp1100:~$ curl -H "X-Forwarded-For: 10.136.0.10" -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache' < Cache-Control: s-maxage=86400, must-revalidate, max-age=0 fabfur@cp1100:~$ curl -H "X-Forwarded-For: 208.80.154.224" -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" https://appservers-rw.discovery.wmnet/wiki/Foobar -v -o /dev/null -s 2>&1 | egrep -i 'expires|cache' # 208.80.154.224 is text-lb.eqiad.wikimedia.org < Expires: Thu, 01 Jan 1970 00:00:00 GMT < Cache-Control: private, must-revalidate, max-age=0
Quick recap: seems that this behavior (uncacheable response) is only reproducible setting 'text-lb.eqiad|codfw.wikimedia.org' XFF IPs (both IPv4 and IPv6) on en.wikipedia.org domain.