Page MenuHomePhabricator

Reduce tool breakage over new ingress-nginx annotation validation rules
Closed, ResolvedPublic

Description

New ingress-nginx versions introduce new validation rules for its annotations, which is breaking some tools using custom Ingress objects (like it did for the interwiki handled in T408570).

P84923 is a list of all tools using custom ingress objects with at least one annotation present, although not all of them are surely affected. In general there are two use cases, redirects and transparent proxying. The redirect case is known to be broken when using placeholders ($1) in the redirect target, the others I haven't tested yet.

(In theory all of this falls into the "using raw Kubernetes features is at your own risk", but I want to at least understand which tools are exactly affected before deciding to break things.)

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Draw the rest of the owltoolforge-repos/containers-redirect!1bd808work/bd808/initialmain
Customize query in GitLab

Event Timeline

I checked and the proxy use case seems to still work fine in Toolsbeta. So I think this is the real list of tools this would affect, generated with P85080:

  • tool-apt/apt-domain:
nginx.ingress.kubernetes.io/temporal-redirect: https://apt-browser.toolforge.org/$1
nginx.ingress.kubernetes.io/use-regex: true
  • tool-capacity-exchange/capacity-exchange-redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://capx.toolforge.org/$1$is_args$args
  • tool-doodle/doodle-redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://wudele.toolforge.org/$1$is_args$args
  • tool-iw-beta/iw-domain:
nginx.ingress.kubernetes.io/temporal-redirect: https://$1.toolforge.org/$3$is_args$args
nginx.ingress.kubernetes.io/use-regex: true
  • tool-iw-beta/iw-root:
nginx.ingress.kubernetes.io/temporal-redirect: https://toolforge.org/$is_args$args
  • tool-jenkins-build-stats/jenkins-build-stats-redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://tools-static.wmflabs.org/jenkins-build-stats/$1$is_args$args
  • tool-keystone-browser/keystone-browser:
nginx.ingress.kubernetes.io/permanent-redirect: https://openstack-browser.toolforge.org/$1
  • tool-libraryupgrader/redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://libraryupgrader2.wmcloud.org/$1$is_args$args
  • tool-toolforge-standards-committee/toolforge-standards-committee-redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_committee$is_args$args
  • tool-toolinfo-scraper/toolinfo-scraper-redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://tools-static.wmflabs.org/toolinfo-scraper/$1$is_args$args
  • tool-vpsalertmanager/vpsalertmanager-domain:
nginx.ingress.kubernetes.io/temporal-redirect: https://prometheus-alerts.wmcloud.org/$1
nginx.ingress.kubernetes.io/use-regex: true
  • tool-wikiconcursos/wikiconcursos-redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://wikiscore.toolforge.org/$1$is_args$args
  • tool-xn--9s9h/xn--9s9h-redirect:
nginx.ingress.kubernetes.io/permanent-redirect: https://meta.wikimedia.org/wiki/User:TheresNoTime/Foxes$is_args$args

I would say this result points to a need for a "simple" way to replace the https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_redirect_to_another_tool_WITHOUT_running_a_webservice recipe. I think if the iw deployments are stripped from the list above the rest are just hostname replacement redirects.

  • tool-keystone-browser/keystone-browser
bd808@tools-bastion-14.tools.eqiad1:~$ become keystone-browser
tools.keystone-browser@tools-bastion-14:~$ toolforge envvars create REDIRECT_HOST
Enter the value of your envvar (Hit Ctrl+C to cancel): openstack-browser.toolforge.org
name           value
REDIRECT_HOST  openstack-browser.toolforge.org
tools.keystone-browser@tools-bastion-14:~$ cat > service.template <<EOF
type: buildservice
buildservice-image: tool-containers/redirect:latest
health-check-path: /healthz
mount: none
cpu: 125m
mem: 128Mi
EOF
tools.keystone-browser@tools-bastion-14:~$ kubectl get ingress
NAME               CLASS    HOSTS                            ADDRESS         PORTS   AGE
keystone-browser   <none>   keystone-browser.toolforge.org   10.111.66.189   80      5y320d
tools.keystone-browser@tools-bastion-14:~$ kubectl delete ingress keystone-browser
ingress.networking.k8s.io "keystone-browser" deleted
tools.keystone-browser@tools-bastion-14:~$ toolforge webservice start
Starting webservice...............
tools.keystone-browser@tools-bastion-14:~$ curl -sI 'https://keystone-browser.toolforge.org/some/path?foo=1&bar=baz' | grep -i location:
location: https://openstack-browser.toolforge.org/some/path?foo=1&bar=baz

I fumbled for a bit with my first service.template. I set mem: 64Mi which kept the container from starting as we have an admission controller rule of some sort that enforces a minimum memory usage: Error creating: pods "keystone-browser-94c5c78c8-zm927" is forbidden: minimum memory usage per Container is 100Mi, but request is 64Mi. This failure case is not well handled by the frontend tooling. webservice status told me that things were running. Finding out that the ReplicaSet was unable to schedule a Pod took a lot of poking around with kubectl. I will try to remember for next time that kubectl events can be a helpful place to find clues.

  • tool-apt/apt-domain:

It looks like @taavi setup tool-containers/redirect for this one.

  • tool-capacity-exchange/capacity-exchange-redirect:

Replaced with tool-containers/redirect

  • tool-doodle/doodle-redirect:

Replaced with tool-containers/redirect

  • tool-iw-beta/iw-domain:
  • tool-iw-beta/iw-root:

Obsolete tool. I disabled it.

  • tool-jenkins-build-stats/jenkins-build-stats-redirect:

This one needs to work a bit differently than the current tool-containers/redirect service. I will circle back here.

  • tool-keystone-browser/keystone-browser:

Replaced with tool-containers/redirect

  • tool-libraryupgrader/redirect:

It looks like @taavi setup tool-containers/redirect for this one.

  • tool-toolforge-standards-committee/toolforge-standards-committee-redirect:

This one needs to work a bit differently than the current tool-containers/redirect service. I will circle back here.

  • tool-toolinfo-scraper/toolinfo-scraper-redirect:

This one needs to work a bit differently than the current tool-containers/redirect service. I will circle back here.

  • tool-vpsalertmanager/vpsalertmanager-domain:

Replaced with tool-containers/redirect

  • tool-wikiconcursos/wikiconcursos-redirect:

Replaced with tool-containers/redirect

  • tool-xn--9s9h/xn--9s9h-redirect:

This one needs to work a bit differently than the current tool-containers/redirect service. I will circle back here.

The remaining jenkins-build-stats, toolforge-standards-committee, toolinfo-scraper, and xn--9s9h tools all need to be able to manipulate the path as well as the host while redirecting. I think that adding an optional REDIRECT_PATH envvar to tool-containers/redirect's configuration will allow the needed configuration.

  • jenkins-build-stats
  • toolforge-standards-committee
  • toolinfo-scraper
  • xn--9s9h

I think that is all of them. @taavi would you mind checking my work to make sure I didn't miss one of them accidentally?

It seems like you left some of the old ingress objects in place. I deleted those, and manually checked that all of those redirects still function. Thank you!!!