Page MenuHomePhabricator

Kubernetes ingress passes it's port & proto to apps rather than the port & proto from the front proxy
Closed, ResolvedPublic

Description

Visible at https://tools.wmflabs.org/bd808-test/si.php:

$_SERVER['HTTP_X_FORWARDED_PROTO']	http
$_SERVER['HTTP_X_FORWARDED_PORT']	8080

I expected to see https and 443 so that apps can inspect these settings when generating canonical URLs. Our proxy in front of a proxy setup is non-standard so it is not unexpected that this is going to require some customization in the nginx ingress.

Event Timeline

bd808 created this task.Dec 22 2019, 4:44 AM

Mentioned in SAL (#wikimedia-cloud) [2019-12-22T04:45:57Z] <bd808> Migrated to PHP 7.3 and new kubernetes cluster. JS and CSS broken due to T241310

Change 560323 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] Toolforge: pass X-Forwared-* headers from front proxy to apps

https://gerrit.wikimedia.org/r/560323

bd808 added a comment.Dec 22 2019, 6:24 PM

Hot patched from tools-k8s-control-1

$ ssh tools-k8s-control-1.tools.eqiad.wmflabs
$ sudo su -
$ kubectl get configmap nginx-configuration -n ingress-nginx -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"nginx-configuration","namespace":"ingress-nginx"}}
  creationTimestamp: "2019-11-07T13:11:19Z"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-configuration
  namespace: ingress-nginx
  resourceVersion: "133776"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/nginx-configuration
  uid: 2879271f-4129-47f5-8b5d-d37ab92aa0ec
$ kubectl edit configmap nginx-configuration -n ingress-nginx
configmap/nginx-configuration edited
$ kubectl get configmap nginx-configuration -n ingress-nginx -o yaml
apiVersion: v1
data:
  use-forwarded-headers: "true"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"nginx-configuration","namespace":"ingress-nginx"}}
  creationTimestamp: "2019-11-07T13:11:19Z"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-configuration
  namespace: ingress-nginx
  resourceVersion: "8769099"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/nginx-configuration
  uid: 2879271f-4129-47f5-8b5d-d37ab92aa0ec

The change took several minutes to be visible at https://tools.wmflabs.org/bd808-test/si.php. This might be related to https://github.com/kubernetes/ingress-nginx/issues/2567 (closed upstream).

After the change applied, $_SERVER['HTTP_X_FORWARDED_PROTO'] is "https" as hoped. $_SERVER['HTTP_X_FORWARDED_PORT'] is still "8080" as we apparently are not setting that header in dynamicprox/urlproxy. I will add that to the patch.

Mentioned in SAL (#wikimedia-cloud) [2019-12-22T18:52:37Z] <bd808> Disabled Puppet on tools-proxy-06.tools.eqiad.wmflabs to test nginx config change (T241310)

Mentioned in SAL (#wikimedia-cloud) [2019-12-22T20:13:50Z] <bd808> Enabled Puppet on tools-proxy-06.tools.eqiad.wmflabs after nginx config test (T241310)

bd808 triaged this task as High priority.Dec 23 2019, 12:34 AM

Change 560323 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Toolforge: pass X-Forwared-* headers from front proxy to apps

https://gerrit.wikimedia.org/r/560323

bd808 closed this task as Resolved.Dec 23 2019, 9:47 PM
bd808 claimed this task.