ingress-nginx 1.0 is out: https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0
We should upgrade toolforge and paws (both deploy it via helm) to it.
ingress-nginx 1.0 is out: https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0
We should upgrade toolforge and paws (both deploy it via helm) to it.
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
toolforge: Update to ingress-nginx v1.0 | operations/puppet | production | +8 -3 | |
kubeadm: add helm-diff | operations/puppet | production | +35 -12 |
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Open | None | T359641 Upgrade Toolforge Kubernetes to version 1.27 | |||
Open | None | T327025 Upgrade Toolforge Kubernetes to version 1.26 | |||
Open | None | T316107 Upgrade Toolforge Kubernetes to version 1.25 | |||
Resolved | aborrero | T307651 Upgrade Toolforge Kubernetes to version 1.24 | |||
Resolved | taavi | T298005 Upgrade Toolforge Kubernetes to version 1.23 | |||
Resolved | taavi | T286856 Upgrade Toolforge Kubernetes to latest 1.22 | |||
Restricted Task | |||||
Resolved | taavi | T292771 upgrade to ingress-nginx 1.0 | |||
Resolved | rook | T291589 Upgrade paws jupyterhub |
Warning, ingress nginx 1.0 will refuse to work with extensions/v1beta1 ingresses regardless of cluster version. @mdipietro and I figured this out experimenting with T291589: Upgrade paws jupyterhub. That should be no issue for most tools per se, as long as it reads existing ingress objects (worth checking if any still exist...they probably do), but jupyterhub 0.9.0 still uses that ingress version.
Toolsbeta should be fine, tools *might* be ok and PAWS just needs jupyterhub upgraded first.
I think networking.k8s.io/v1beta1 might be ok...just definitely not extensions/v1beta1.
We deploy ingress-nginx via Helm, as of right now it's the only component deployed with Helm and not been upgraded since we moved it to Helm, making this a fully new procedure. Without having looked very far into this, I think we have a few major issues with our Helm tooling, namely no way of diffing the changes and not having very good control of what version we are upgrading to.
The current documentation just advises to run a blind upgrade command. In theory that's fine, but we probably want to modify it to upgrade to a specific version (and instructions how to look up the latest version). Using helmfile (which is already packaged on apt.wm.o) as of right now is probably overkill, but worth considering if we deploy more components via Helm. Diffs are an easier problem, there's a plugin (helm-diff) that wiki production uses too and should be just be a matter of installing a relevant package.
Change 729577 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] kubeadm: add helm-diff
Change 729577 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] kubeadm: add helm-diff
Mentioned in SAL (#wikimedia-cloud) [2021-10-25T14:33:16Z] <majavah> copy nginx-ingress controller v1.0.4 to internal registry T292771
Mentioned in SAL (#wikimedia-cloud) [2021-10-25T14:41:08Z] <majavah> deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image T292771
Full chart diff is on P17593 ingress-nginx update
Startup logs show this:
I1025 14:50:07.894107 8 store.go:367] "Ignoring ingress because of error while validating ingress class" ingress="tool-fourohfour/fourohfour" error="ingress does not contain a valid IngressClass" I1025 14:50:07.894844 8 store.go:367] "Ignoring ingress because of error while validating ingress class" ingress="tool-fourohfour/fourohfour-subdomain" error="ingress does not contain a valid IngressClass" I1025 14:50:07.895218 8 store.go:367] "Ignoring ingress because of error while validating ingress class" ingress="tool-test/test-subdomain" error="ingress does not contain a valid IngressClass"
Ingress classes are a new resource in K8s 1.19+ and in ingress-nginx v1.0+.
There are two ways to fix this:
The second option is more preferrable in my opinion but it doesn't seem to be backwards compatible. So I'll likely do both at first, then get all existing ingresses updated and then set watchIngressWithoutClass: false. Note that apparently you can set ingressClassName on an ingress even if the class does not exist, but the current ingress controller will not like you.
Change 734294 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] toolforge: Update to ingress-nginx v1.0
Change 734294 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: Update to ingress-nginx v1.0
Mentioned in SAL (#wikimedia-cloud) [2021-10-26T12:11:36Z] <majavah> deploy ingress-nginx v1.0.4 / chart v4.0.6 on toolforge T292771
Mentioned in SAL (#wikimedia-cloud) [2021-11-01T12:31:29Z] <majavah> upgrade ingress-nginx T292771