After adding the restbase-ssl service https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534462/ in the context of T210411 we have noticed that all ProxyFetch checks were failing:
Sep 09 09:59:39 lvs1016 pybal[22763]: [restbase-https_7443 ProxyFetch] WARN: restbase1019.eqiad.wmnet (disabled/partially up/not pooled): Fetch failed (https://localhost/), 5.419 s
The reason for this is that, according to ats-tls logs on a upload node, PyBal's ProxyFetch checks use HTTP/1.0 when factory.scheme is set to https:
Date:2019-09-09 Time:09:40:38 Client-IP:2001:df2:e500:101:10:132:0:12 HTTPVersion:HTTP/1.0 SSL:1 SSLVersion:TLSv1.2 SSLCipher:ECDHE-ECDSA-CHACHA20-POLY1305 SSLCurve:X25519 ReqMethod:GET RespStatus:200 OriginStatus:200 ReqURL:https://varnishcheck.wikimedia.org/from/pybal BereqURL:GET http://127.0.0.2:3120/from/pybal HTTP/1.0 ReqHeader:User-Agent:Twisted PageGetter ReqHeader:Host:varnishcheck.wikimedia.org
When the check uses plain HTTP, the protocol used is HTTP/1.1 instead:
127.0.0.1 - - [09/Sep/2019:09:50:27 +0000] "GET http://varnishcheck.wikimedia.org/from/pybal HTTP/1.1" 200 40 "-" "Twisted PageGetter"
By default, envoy responds with 426 to HTTP/1.0 requests, making the ProxyFetch checks fail. We can instruct envoy not to do so by setting accept_http_10, but still PyBal's behavior should be fixed.