Page MenuHomePhabricator

toolsbeta: ingress admission controller doesn't accept valid FQDN patterns
Closed, ResolvedPublic

Description

Something is weird with the ingress admission controller in toolsbeta. File:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    # Enable client certificate authentication
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    # Create the secret containing the trusted ca certificates
    # TODO: what to do with this
    nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret"
    # Specify the verification depth in the client certificates chain
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    # Specify an error page to be redirected to verification errors
    nginx.ingress.kubernetes.io/auth-tls-error-page: "https://jobs.toolsbeta.wmflabs.org/"
    # Specify if certificates are passed to upstream server
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
  name: toolforge-jobs-api
  namespace: ingress-nginx-jobs
spec:
  rules:
  - host: jobs.toolsbeta.wmflabs.org
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 8080
        path: /

Error message (I tried a couple of variants):

root@toolsbeta-test-k8s-control-4:~# grep wmflabs ingress.yaml 
    nginx.ingress.kubernetes.io/auth-tls-error-page: "https://jobs.toolsbeta.wmflabs.org/"
  - host: jobs.toolsbeta.wmflabs.org
root@toolsbeta-test-k8s-control-4:~# kubectl apply -f ingress.yaml 
Error from server: error when creating "ingress.yaml": admission webhook "ingress-admission.tools.wmflabs.org" denied the request: Ingress host must be <toolname>.toolforge.org or tools.wmflabs.org/<toolname>
root@toolsbeta-test-k8s-control-4:~# egrep toolforge\|wmflabs ingress.yaml 
    nginx.ingress.kubernetes.io/auth-tls-error-page: "https://jobs.toolsbeta.wmflabs.org/"
  - host: jobs.toolforge.org
root@toolsbeta-test-k8s-control-4:~# kubectl apply -f ingress.yaml 
Error from server: error when creating "ingress.yaml": admission webhook "ingress-admission.tools.wmflabs.org" denied the request: Ingress host must be <toolname>.toolforge.org or tools.wmflabs.org/<toolname>
root@toolsbeta-test-k8s-control-4:~# kubectl logs pod/ingress-admission-98cf75f59-8mrfx -n ingress-admission
time="2021-02-19T12:42:09Z" level=info msg="&{0.0.0.0:8080 /etc/webhook/certs/cert.pem /etc/webhook/certs/key.pem [toolforge.org wmflabs.org wmcloud.org toolsbeta.wmflabs.org toolsbeta.wmcloud.org] false}"

This is despite https://gerrit.wikimedia.org/r/c/cloud/toolforge/ingress-admission-controller/+/663230 being deployed already.

I guess the error message is misleading, and what's failing is other stuff like the namespace pattern, or the metadata name, or the like.

Event Timeline

aborrero renamed this task from toolsbeta: ingress admission controller doesn't accept valid domains to toolsbeta: ingress admission controller doesn't accept valid FQDN patterns.Feb 23 2021, 10:42 AM
aborrero created this task.
aborrero triaged this task as Medium priority.Feb 23 2021, 10:46 AM

Did we rebuild the image after the last change?

Either way, the error looks misleading :)

Did we rebuild the image after the last change?

To answer myself, yes, it looks that way because the image is only 4 days old. Must be buried in the checks. I was hoping it would be simpler.

The initial message has namespace: ingress-nginx-jobs, so isn't the ingress controller doing what it's supposed to do? Unless the namespace is tool-fourohfour or kube-system, the namespace minus the first five chars (effectively removing tool-) must match the first part of the domain, so for jobs.toolsbeta.wmflabs.org it should be tool-jobs or similar and for the current ingress-nginx-jobs it would only accept ss-nginx-jobs.toolsbeta.wmflabs.org.

The initial message has namespace: ingress-nginx-jobs, so isn't the ingress controller doing what it's supposed to do? Unless the namespace is tool-fourohfour or kube-system, the namespace minus the first five chars (effectively removing tool-) must match the first part of the domain, so for jobs.toolsbeta.wmflabs.org it should be tool-jobs or similar and for the current ingress-nginx-jobs it would only accept ss-nginx-jobs.toolsbeta.wmflabs.org.

Circling back to this, I just set it to do debug logging, but you are likely entirely correct here. Ingresses outside of tools must either be in kube-system or be defined by their namespace. We should exempt the ingress controller from checking the ingress-nginx-jobs namespace to fix this since it isn't a tool-jobs tool.

Turned off debug logging and instead, I think I'll set up a kustomize setup on the controller so that you can have toolsbeta have different values in service.yaml like I have on maintain-kubeusers. That will prevent people from overwriting a manual change to the service settings.

Change 675890 had a related patch set uploaded (by Bstorm; author: Bstorm):

[cloud/toolforge/ingress-admission-controller@master] add jobs service namespace

https://gerrit.wikimedia.org/r/675890

There's no harm in setting that namespace as exempt in all deployments really. Adding kustomize is a bit quirky because of how I hacked in the ca-client-file in, so I don't want this blocked by it while I figure out the right way to change that bit.

Change 675926 had a related patch set uploaded (by Bstorm; author: Bstorm):

[cloud/toolforge/ingress-admission-controller@master] kustomize the deployment to allow more flexibility

https://gerrit.wikimedia.org/r/675926

Change 675890 abandoned by Bstorm:

[cloud/toolforge/ingress-admission-controller@master] add jobs service namespace

Reason:

I88458b981648b036d7e14061e67 seems better

https://gerrit.wikimedia.org/r/675890

I think that will fix everything up. If we are deploying ingresses for things that aren't "tools" in the strict sense of the word, they should be in a safelisted namespace.

The initial message has namespace: ingress-nginx-jobs, so isn't the ingress controller doing what it's supposed to do? Unless the namespace is tool-fourohfour or kube-system, the namespace minus the first five chars (effectively removing tool-) must match the first part of the domain, so for jobs.toolsbeta.wmflabs.org it should be tool-jobs or similar and for the current ingress-nginx-jobs it would only accept ss-nginx-jobs.toolsbeta.wmflabs.org.

You are 100% right. Sorry I overlook it, and then moved on to other stuff and stopped paying attention.

Change 675926 merged by jenkins-bot:

[cloud/toolforge/ingress-admission-controller@master] kustomize the deployment to allow more flexibility

https://gerrit.wikimedia.org/r/675926

Mentioned in SAL (#wikimedia-cloud) [2021-03-31T18:22:13Z] <bstorm> redeploy ingress-admission controller with kubectl apply -k deploys/toolsbeta from the repo T275478

aborrero reassigned this task from aborrero to Bstorm.