cumin1002:~$ curl -H 'Host: trace.wikimedia.org' https://jaeger-query.svc.eqiad.wmnet:30443 -v
* Uses proxy env variable no_proxy == '.wmnet'
*   Trying 10.2.2.78:30443...
* Connected to jaeger-query.svc.eqiad.wmnet (10.2.2.78) port 30443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=jaeger-collector-http.discovery.wmnet
*  start date: Feb  4 14:43:00 2024 GMT
*  expire date: Mar  3 14:43:00 2024 GMT
*  subjectAltName: host "jaeger-query.svc.eqiad.wmnet" matched cert's "jaeger-query.svc.eqiad.wmnet"
*  issuer: C=US; L=San Francisco; O=Wikimedia Foundation, Inc; OU=SRE Foundations; CN=discovery
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564c76982620)
> GET / HTTP/2
> Host: trace.wikimedia.org
> user-agent: curl/7.74.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 503 
< content-length: 114
< content-type: text/plain
< date: Thu, 15 Feb 2024 13:57:48 GMT
< server: istio-envoy
< 
* Connection #0 to host jaeger-query.svc.eqiad.wmnet left intact
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure

root@deploy2002:~# kubectl logs -l app=istio-ingressgateway -n istio-system --all-containers -f  | grep trace.wikimedia
{"response_flags":"UF,URX","connection_termination_details":null,"authority":"trace.wikimedia.org","response_code":503,"method":"GET","response_code_details":"upstream_reset_before_response_started{connection_failure}","downstream_remote_address":"10.192.32.7:38402","duration":30019,"downstream_local_address":"10.67.83.217:8443","upstream_transport_failure_reason":null,"protocol":"HTTP/2","request_id":"35d55465-1117-426a-a0db-062c43f91d5e","path":"/","upstream_cluster":"out
bound|16686||main-jaeger-query.jaeger.svc.cluster.local","user_agent":"curl/7.64.0","x_forwarded_for":"10.192.32.7","requested_server_name":"jaeger-collector-http.svc.eqiad.wmnet","upstream_host":"10.67.83.197:4180","bytes_received":0,"bytes_sent":114,"route_name":"default-route","upstream_service_time":null,"start_time":"2024-02-16T10:26:42.341Z","upstream_local_address":null}

{
  "response_flags": "UF,URX",
  "connection_termination_details": null,
  "authority": "trace.wikimedia.org",
  "response_code": 503,
  "method": "GET",
  "response_code_details": "upstream_reset_before_response_started{connection_failure}",
  "downstream_remote_address": "10.192.32.7:38402",
  "duration": 30019,
  "downstream_local_address": "10.67.83.217:8443",
  "upstream_transport_failure_reason": null,
  "protocol": "HTTP/2",
  "request_id": "35d55465-1117-426a-a0db-062c43f91d5e",
  "path": "/",
  "upstream_cluster": "outbound|16686||main-jaeger-query.jaeger.svc.cluster.local",
  "user_agent": "curl/7.64.0",
  "x_forwarded_for": "10.192.32.7",
  "requested_server_name": "jaeger-collector-http.svc.eqiad.wmnet",
  "upstream_host": "10.67.83.197:4180",
  "bytes_received": 0,
  "bytes_sent": 114,
  "route_name": "default-route",
  "upstream_service_time": null,
  "start_time": "2024-02-16T10:26:42.341Z",
  "upstream_local_address": null
}

filippo@deploy2002:~$ kubectl logs -l app.kubernetes.io/component=query --all-containers -f
[2024/02/09 17:31:52] [provider.go:55] Performing OIDC Discovery...
[2024/02/09 17:31:53] [proxy.go:89] mapping path "/" => upstream "https://localhost:16686"
[2024/02/09 17:31:53] [oauthproxy.go:166] OAuthProxy configured for OpenID Connect Client ID: jaeger-ui
[2024/02/09 17:31:53] [oauthproxy.go:172] Cookie settings: name:_oauth2_proxy secure(https):true httponly:true expiry:168h0m0s domains:trace.wikimedia.org path:/ samesite: refresh:disabled
{"level":"info","ts":1707499912.824807,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2 SubChannel #3] Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
{"level":"info","ts":1707499912.8248394,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2 SubChannel #3] Subchannel picks a new address \":16685\" to connect","system":"grpc","grpc_log":true}
{"level":"info","ts":1707499912.8251963,"caller":"app/server.go:282","msg":"Starting HTTP server","port":16686,"addr":":16686"}
{"level":"info","ts":1707499912.825271,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc000525338, {CONNECTING <nil>}","system":"grpc","grpc_log":true}
{"level":"info","ts":1707499912.8253164,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
{"level":"info","ts":1707499912.826207,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2 SubChannel #3] Subchannel Connectivity change to READY","system":"grpc","grpc_log":true}
{"level":"info","ts":1707499912.8262506,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc000525338, {READY <nil>}","system":"grpc","grpc_log":true}
{"level":"info","ts":1707499912.8262668,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] Channel Connectivity change to READY","system":"grpc","grpc_log":true}
{"level":"warn","ts":1707865331.3160307,"caller":"tlscfg/cert_watcher.go:92","msg":"Certificate has been removed, using the last known version","certificate":"/tls/tls.key"}
{"level":"warn","ts":1707865331.3167005,"caller":"tlscfg/cert_watcher.go:92","msg":"Certificate has been removed, using the last known version","certificate":"/tls/tls.crt"}

taavi@deploy2002 ~ <jaeger/aux-k8s-eqiad> $ kubectl port-forward main-jaeger-query-7f56949ddd-fdl57 4180 &
[1] 14693
taavi@deploy2002 ~ <jaeger/aux-k8s-eqiad> $ Forwarding from 127.0.0.1:4180 -> 4180
Forwarding from [::1]:4180 -> 4180

taavi@deploy2002 ~ <jaeger/aux-k8s-eqiad> $ 
taavi@deploy2002 ~ <jaeger/aux-k8s-eqiad> $ curl localhost:4180
Handling connection for 4180
<a href="https://idp.wikimedia.org/oidc/oidcAuthorize?approval_prompt=force&amp;client_id=jaeger-ui&amp;code_challenge=4t9neTN35X1xh7ImqSgc1Wvm3sY503jsbA_pv0Cnt_xFtwO407_9qV8bCjXy7JEarG4VF9ecMIpD5f-ELJuG0qIb67DaMTOc&amp;code_challenge_method=plain&amp;redirect_uri=https%3A%2F%2Ftrace.wikimedia.org%2Foauth2%2Fcallback&amp;response_type=code&amp;scope=openid+email+profile&amp;state=rl7CT7a3nIf5WPk4SqeZQ7ITcWGS30MdBtImUHq4e6E%3A%2F">Found</a>.

taavi@deploy2002 ~ <jaeger/aux-k8s-eqiad> $ curl https://localhost:4180
Handling connection for 4180
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong versE0216 10:54:01.773755   14693 portforward.go:391] error copying from local connection to remote stream: read tcp6 [::1]:4180->[::1]:48912: read: connection reset by peer

$ curl -v https://localhost:4180 -k
* Uses proxy env variable no_proxy == 'wikipedia.org,wikimedia.org,wikibooks.org,wikinews.org,wikiquote.org,wikisource.org,wikiversity.org,wikivoyage.org,wikidata.org,wikiworkshop.org,wikifunctions.org,wiktionary.org,mediawiki.org,wmfusercontent.org,w.wiki,wmnet,127.0.0.1,::1'
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4180 (#0)
Handling connection for 4180
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=main-jaeger-query.jaeger.svc
*  start date: Feb 13 22:57:00 2024 GMT
*  expire date: Mar 12 22:57:00 2024 GMT
*  issuer: C=US; L=San Francisco; O=Wikimedia Foundation, Inc; OU=SRE Foundations; CN=discovery
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost:4180
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 302 Found
< Cache-Control: no-cache, no-store, must-revalidate, max-age=0
< Content-Type: text/html; charset=utf-8
< Expires: Thu, 01 Jan 1970 00:00:00 UTC
< Location: https://idp.wikimedia.org/oidc/oidcAuthorize?approval_prompt=force&client_id=jaeger-ui&code_challenge=GI9kWOsnmCH9eAn1.pIn5y-CK~kA9-ww7fNqDQioOqBPWr~W326MMTau0516TdRO7Kkgg9X.XjvsICtk1j~yz14p7_ayD5Kk&code_challenge_method=plain&redirect_uri=https%3A%2F%2Ftrace.wikimedia.org%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=K-txYNvO8X8pnUxRfq_KHzc-7C9L0OLUR_y1UYuJhGw%3A%2F
< Set-Cookie: _oauth2_proxy_csrf=4E5nwGoKYxUNdK3GxPXG9zznBL54eoTc0Vf7WHSJ20RDQ6a7reeWrwheGFn87Liu0GAsxCVM534p7u72EnGoxH37MALtnazEPB0SnIsMgT1h6VTGvjslmZbubIdFcPKktHAaDwD4kHEH4SCOW3nX1uzDTcN7LbtpTN_K-ukjh0B7EA9X9n4opKQbo8y0wzE1rNyMtJrkIT19u115eCmY_BZ8SceDTTHaN23U5hp8s1F_mFrnL_tEWzqibW8wGg==|1708435326|Lwh_x1KQ7l4DLjgcH6UMpk62k5Y1w_L9IQyAJGX8Wt8=; Path=/; Domain=trace.wikimedia.org; Expires=Tue, 20 Feb 2024 13:37:06 GMT; HttpOnly; Secure
< X-Accel-Expires: 0
< Date: Tue, 20 Feb 2024 13:22:06 GMT
< Content-Length: 446
< 
<a href="https://idp.wikimedia.org/oidc/oidcAuthorize?approval_prompt=force&amp;client_id=jaeger-ui&amp;code_challenge=GI9kWOsnmCH9eAn1.pIn5y-CK~kA9-ww7fNqDQioOqBPWr~W326MMTau0516TdRO7Kkgg9X.XjvsICtk1j~yz14p7_ayD5Kk&amp;code_challenge_method=plain&amp;redirect_uri=https%3A%2F%2Ftrace.wikimedia.org%2Foauth2%2Fcallback&amp;response_type=code&amp;scope=openid+email+profile&amp;state=K-txYNvO8X8pnUxRfq_KHzc-7C9L0OLUR_y1UYuJhGw%3A%2F">Found</a>.

* Connection #0 to host localhost left intact

taavi@deploy2002 ~ $ curl -v --connect-to ::jaeger-query.svc.eqiad.wmnet https://trace.wikimedia.org:30443
* Uses proxy env variable no_proxy == 'wikipedia.org,wikimedia.org,wikibooks.org,wikinews.org,wikiquote.org,wikisource.org,wikiversity.org,wikivoyage.org,wikidata.org,wikiworkshop.org,wikifunctions.org,wiktionary.org,mediawiki.org,wmfusercontent.org,w.wiki,wmnet,127.0.0.1,::1'
* Connecting to hostname: jaeger-query.svc.eqiad.wmnet
*   Trying 10.2.2.78...
* TCP_NODELAY set
* Connected to jaeger-query.svc.eqiad.wmnet (10.2.2.78) port 30443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=jaeger-collector-http.discovery.wmnet
*  start date: Feb  4 14:43:00 2024 GMT
*  expire date: Mar  3 14:43:00 2024 GMT
*  subjectAltName does not match trace.wikimedia.org
* SSL: no alternative certificate subject name matches target host name 'trace.wikimedia.org'
* Closing connection 0
curl: (60) SSL: no alternative certificate subject name matches target host name 'trace.wikimedia.org'
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.