Page MenuHomePhabricator
Create Task
Maniphest T241008

New k8s cluster routing behaving strangely for bd808-test tool
Closed, ResolvedPublic
Actions

Description

I switched my bd808-test tool over to the new cluster as a test case. The switch went fine and the pod is running, but the ingress routing is acting strange. All URL paths under https://tools.wmflabs.org/bd808-test/ seem to be routed to the "/" route in the pod which is currently a static html page.

Could the k8s ingress be mapping all URLs under /bd808-test/ to / when it proxies to the pod?

Even more weird/scary, requests for these non-existent tools also get mapped to https://tools.wmflabs.org/bd808-test/index.html:

Details

SubjectRepoBranchLines +/-
Preserve tool name and path info in k8s ingress rewriteoperations/software/tools-webservicemaster+6 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

bd808 created this task.Dec 17 2019, 10:43 PM
bd808 added a comment.Dec 17 2019, 10:57 PM
$ kubectl get ingress
NAME         HOSTS               ADDRESS   PORTS   AGE
bd808-test   tools.wmflabs.org             80      15m
$ kubectl describe ingress bd808-test
Name:             bd808-test
Namespace:        tool-bd808-test
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  tools.wmflabs.org
                     /bd808-test   bd808-test:8000 (192.168.29.6:8000)
Annotations:
  nginx.ingress.kubernetes.io/rewrite-target:  /
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  16m   nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  16m   nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  16m   nginx-ingress-controller  Ingress tool-bd808-test/bd808-test

I think that this Ingress rule should actually look something like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  labels:
    name: bd808-test
    toolforge: tool
    tools.wmflabs.org/webservice: "true"
    tools.wmflabs.org/webservice-version: "1"
  name: bd808-test
  namespace: tool-bd808-test
spec:
  rules:
  - host: tools.wmflabs.org
    http:
      paths:
      - backend:
          serviceName: bd808-test
          servicePort: 8000
        path: /bd808-test(/|$)(.*)
Stashbot added a comment.Dec 17 2019, 10:57 PM

Mentioned in SAL (#wikimedia-cloud) [2019-12-17T22:57:32Z] <bd808> Manually updated ingress rules (T241008)

bd808 added a comment.Dec 17 2019, 10:59 PM
In T241008#5749502, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2019-12-17T22:57:32Z] <bd808> Manually updated ingress rules (T241008)

Using kubectl edit ingress bd808-test to make the changes I imagined above fixes both the internal app links and the problem of capturing too many toolnames. Now to make a patch for webservice to adjust that for everyone.

gerritbot added a comment.Dec 17 2019, 11:17 PM

Change 558726 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Preserve tool name and path info in k8s ingress rewrite

https://gerrit.wikimedia.org/r/558726

gerritbot added a project: Patch-For-Review.Dec 17 2019, 11:17 PM
aborrero assigned this task to bd808.Dec 18 2019, 10:29 AM
aborrero triaged this task as High priority.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Thanks for working on this!

Please, update the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_ingress#ingress_objects

Also, worth double-checking that our ingress admission controller allows the new setting: https://github.com/wikimedia/cloud-toolforge-ingress-admission-controller/blob/master/server/ingressadmission.go

gerritbot added a comment.Dec 18 2019, 3:31 PM

Change 558726 merged by Bstorm:
[operations/software/tools-webservice@master] Preserve tool name and path info in k8s ingress rewrite

https://gerrit.wikimedia.org/r/558726

bd808 closed this task as Resolved.Dec 18 2019, 5:53 PM

Testing after @Bstorm built and deployed a new webservice package:

$ webservice stop
$ kubectl describe ingress bd808-test
Error from server (NotFound): ingresses.extensions "bd808-test" not found
$ webservice --backend=kubernetes php7.2 start
Starting webservice...
$ kubectl describe ingress bd808-test
Name:             bd808-test
Namespace:        tool-bd808-test
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  tools.wmflabs.org
                     /bd808-test(/|$)(.*)   bd808-test:8000 (192.168.29.7:8000)
Annotations:
  nginx.ingress.kubernetes.io/rewrite-target:  /bd808-test/$2
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  8s    nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  8s    nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
  Normal  CREATE  8s    nginx-ingress-controller  Ingress tool-bd808-test/bd808-test
bd808 mentioned this in T241525: fourohfour tool returning "replag" as the assigned tool for all 404 urls.Dec 29 2019, 2:28 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL