Visible at https://tools.wmflabs.org/bd808-test/si.php:
$_SERVER['HTTP_X_FORWARDED_PROTO'] http $_SERVER['HTTP_X_FORWARDED_PORT'] 8080
I expected to see https and 443 so that apps can inspect these settings when generating canonical URLs. Our proxy in front of a proxy setup is non-standard so it is not unexpected that this is going to require some customization in the nginx ingress.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Toolforge: pass X-Forwared-* headers from front proxy to apps | operations/puppet | production | +6 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | • Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
| Restricted Task | |||||
| Resolved | bd808 | T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail | |||
| Resolved | • Bstorm | T236565 "tools" Cloud VPS project jessie deprecation | |||
| Resolved | aborrero | T101651 Set up toolsbeta more fully to help make testing easier | |||
| Resolved | • Bstorm | T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf) | |||
| Resolved | • Bstorm | T246059 Add admin account creation to maintain-kubeusers | |||
| Resolved | • Bstorm | T154504 Make webservice backend default to kubernetes | |||
| Declined | None | T245230 Investigate cpu/ram requests and limits for DaemonSets pods | |||
| Resolved | • Bstorm | T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster | |||
| Resolved | bd808 | T241310 Kubernetes ingress passes it's port & proto to apps rather than the port & proto from the front proxy |
Mentioned in SAL (#wikimedia-cloud) [2019-12-22T04:45:57Z] <bd808> Migrated to PHP 7.3 and new kubernetes cluster. JS and CSS broken due to T241310
Change 560323 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] Toolforge: pass X-Forwared-* headers from front proxy to apps
Hot patched from tools-k8s-control-1
$ ssh tools-k8s-control-1.tools.eqiad.wmflabs $ sudo su - $ kubectl get configmap nginx-configuration -n ingress-nginx -o yaml apiVersion: v1 kind: ConfigMap metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"nginx-configuration","namespace":"ingress-nginx"}} creationTimestamp: "2019-11-07T13:11:19Z" labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx name: nginx-configuration namespace: ingress-nginx resourceVersion: "133776" selfLink: /api/v1/namespaces/ingress-nginx/configmaps/nginx-configuration uid: 2879271f-4129-47f5-8b5d-d37ab92aa0ec $ kubectl edit configmap nginx-configuration -n ingress-nginx configmap/nginx-configuration edited $ kubectl get configmap nginx-configuration -n ingress-nginx -o yaml apiVersion: v1 data: use-forwarded-headers: "true" kind: ConfigMap metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"nginx-configuration","namespace":"ingress-nginx"}} creationTimestamp: "2019-11-07T13:11:19Z" labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx name: nginx-configuration namespace: ingress-nginx resourceVersion: "8769099" selfLink: /api/v1/namespaces/ingress-nginx/configmaps/nginx-configuration uid: 2879271f-4129-47f5-8b5d-d37ab92aa0ec
The change took several minutes to be visible at https://tools.wmflabs.org/bd808-test/si.php. This might be related to https://github.com/kubernetes/ingress-nginx/issues/2567 (closed upstream).
After the change applied, $_SERVER['HTTP_X_FORWARDED_PROTO'] is "https" as hoped. $_SERVER['HTTP_X_FORWARDED_PORT'] is still "8080" as we apparently are not setting that header in dynamicprox/urlproxy. I will add that to the patch.
Mentioned in SAL (#wikimedia-cloud) [2019-12-22T18:52:37Z] <bd808> Disabled Puppet on tools-proxy-06.tools.eqiad.wmflabs to test nginx config change (T241310)
Mentioned in SAL (#wikimedia-cloud) [2019-12-22T20:13:50Z] <bd808> Enabled Puppet on tools-proxy-06.tools.eqiad.wmflabs after nginx config test (T241310)
Change 560323 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Toolforge: pass X-Forwared-* headers from front proxy to apps