Page MenuHomePhabricator

New k8s cluster routing behaving strangely for bd808-test tool
Closed, ResolvedPublic

Description

I switched my bd808-test tool over to the new cluster as a test case. The switch went fine and the pod is running, but the ingress routing is acting strange. All URL paths under https://tools.wmflabs.org/bd808-test/ seem to be routed to the "/" route in the pod which is currently a static html page.

Could the k8s ingress be mapping all URLs under /bd808-test/ to / when it proxies to the pod?

Even more weird/scary, requests for these non-existent tools also get mapped to https://tools.wmflabs.org/bd808-test/index.html:

Event Timeline

bd808 created this task.Dec 17 2019, 10:43 PM
$ kubectl get ingress
NAME         HOSTS               ADDRESS   PORTS   AGE
bd808-test   tools.wmflabs.org             80      15m
$ kubectl describe ingress bd808-test
Name:             bd808-test
Namespace:        tool-bd808-test
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  tools.wmflabs.org
                     /bd808-test   bd808-test:8000 (192.168.29.6:8000)
Annotations:
  nginx.ingress.kubernetes.io/rewrite-target:  /
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  16m   nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  16m   nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  16m   nginx-ingress-controller  Ingress tool-bd808-test/bd808-test

I think that this Ingress rule should actually look something like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  labels:
    name: bd808-test
    toolforge: tool
    tools.wmflabs.org/webservice: "true"
    tools.wmflabs.org/webservice-version: "1"
  name: bd808-test
  namespace: tool-bd808-test
spec:
  rules:
  - host: tools.wmflabs.org
    http:
      paths:
      - backend:
          serviceName: bd808-test
          servicePort: 8000
        path: /bd808-test(/|$)(.*)

Mentioned in SAL (#wikimedia-cloud) [2019-12-17T22:57:32Z] <bd808> Manually updated ingress rules (T241008)

Mentioned in SAL (#wikimedia-cloud) [2019-12-17T22:57:32Z] <bd808> Manually updated ingress rules (T241008)

Using kubectl edit ingress bd808-test to make the changes I imagined above fixes both the internal app links and the problem of capturing too many toolnames. Now to make a patch for webservice to adjust that for everyone.

Change 558726 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Preserve tool name and path info in k8s ingress rewrite

https://gerrit.wikimedia.org/r/558726

aborrero assigned this task to bd808.Dec 18 2019, 10:29 AM
aborrero triaged this task as High priority.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Thanks for working on this!

Please, update the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_ingress#ingress_objects

Also, worth double-checking that our ingress admission controller allows the new setting: https://github.com/wikimedia/cloud-toolforge-ingress-admission-controller/blob/master/server/ingressadmission.go

Change 558726 merged by Bstorm:
[operations/software/tools-webservice@master] Preserve tool name and path info in k8s ingress rewrite

https://gerrit.wikimedia.org/r/558726

bd808 closed this task as Resolved.Dec 18 2019, 5:53 PM

Testing after @Bstorm built and deployed a new webservice package:

$ webservice stop
$ kubectl describe ingress bd808-test
Error from server (NotFound): ingresses.extensions "bd808-test" not found
$ webservice --backend=kubernetes php7.2 start
Starting webservice...
$ kubectl describe ingress bd808-test
Name:             bd808-test
Namespace:        tool-bd808-test
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  tools.wmflabs.org
                     /bd808-test(/|$)(.*)   bd808-test:8000 (192.168.29.7:8000)
Annotations:
  nginx.ingress.kubernetes.io/rewrite-target:  /bd808-test/$2
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  8s    nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  8s    nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  8s    nginx-ingress-controller  Ingress tool-bd808-test/bd808-test