The process of upgrading nodes is fairly long and tedious. While the control plane should be done with manual care, one node at a time, the worker nodes should probably have a much more automated process:
All worker nodes should have kubeadm upgraded to 1.16.9 before beginning. Refer to https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
- Drain with kubectl drain --force --ignore-daemonsets --delete-local-data $node
- kubeadm upgrade node on the node
- Upgrade kubelet, docker, containerd.io
- Restart docker and kubelet
- kubectl uncordon $node
The process should be done either one-at-a-time or in small batches to avoid overloading nodes during the depooling. It may be possible to encode this all in a spicerack cookbook? It may just be a small script run from a local machine.