v1.15 is going to fall off patch support soon. We need to upgrade to v1.16 ASAP.
There will be several subtasks for this because to do the upgrade we need to fix up webservice, a mistakened use of a deprecated API in maintain-kubeusers and likely other things.
This will start support for ipv6, but support for ipv6 will be far better in 1.18 (now released).
- Update maintain-kubeusers to use the policy apiGroup instead of extensions
- T197930: Replace pykube with a library that works with newer Kubernetes APIs
- Investigate and actually T250863: Upgrade calico to a more recent version (current is 3.14.0) since we are getting behind there. If this ends up better rolled into the k8s upgrade, do the update further down the list.
- T250867: Script the process of upgrading a node with kubeadm to 1.16.9 because doing it 50+ times by hand sounds bad.
- T250866: Stage packages for upstream kubeadm v1.16.9 to use in Toolforge
- Install kubectl and kubeadm first in toolsbeta to validate that they work correctly with the 1.15 cluster.
- Follow the procedure in https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ to upgrade toolsbeta to 1.16.9
- This implies upgrading the control plane nodes by draining them one at a time and upgrading. kubelet is a package upgrade, the rest is kubeadm
- When the control plane is fully upgraded, then the worker nodes get upgraded, again draining them, upgrade kubeadm, run the upgrade, then upgrade kubelet. The procedure is very detailed in the doc.
- Upgrade tools after announcements
- Refresh all external certs (Prometheus, admission controllers) T250874