Upgrade procedure: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes
Refer to the link above for the detailed procedure, and update the checkboxes as you complete them.
If multiple people are working on the upgrade, you can copy the checklist to an Etherpad for easier collaborative editing.
Use this command from a toolforge control node to quickly generate a list of nodes:
for node in $(kubectl get nodes -o json | jq '.items[].metadata.name' -r); do echo " - [] $node"; done
- Run functional tests
- Add a silence in alertmanager
- Update IRC topic (only for "tools" cluster)
- Run prepare_upgrade cookbook
- Upgrade control nodes
- tools-k8s-control-7
- tools-k8s-control-8
- tools-k8s-control-9
- Upgrade worker nodes
- tools-k8s-worker-102
- tools-k8s-worker-103
- tools-k8s-worker-105
- tools-k8s-worker-106
- tools-k8s-worker-107
- tools-k8s-worker-108
- tools-k8s-worker-109
- tools-k8s-worker-110
- tools-k8s-worker-111
- tools-k8s-worker-nfs-1
- tools-k8s-worker-nfs-10
- tools-k8s-worker-nfs-11
- tools-k8s-worker-nfs-12
- tools-k8s-worker-nfs-13
- tools-k8s-worker-nfs-14
- tools-k8s-worker-nfs-16
- tools-k8s-worker-nfs-17
- tools-k8s-worker-nfs-19
- tools-k8s-worker-nfs-2
- tools-k8s-worker-nfs-21
- tools-k8s-worker-nfs-22
- tools-k8s-worker-nfs-23
- tools-k8s-worker-nfs-24
- tools-k8s-worker-nfs-26
- tools-k8s-worker-nfs-27
- tools-k8s-worker-nfs-3
- tools-k8s-worker-nfs-32
- tools-k8s-worker-nfs-33
- tools-k8s-worker-nfs-34
- tools-k8s-worker-nfs-35
- tools-k8s-worker-nfs-36
- tools-k8s-worker-nfs-37
- tools-k8s-worker-nfs-38
- tools-k8s-worker-nfs-39
- tools-k8s-worker-nfs-40
- tools-k8s-worker-nfs-41
- tools-k8s-worker-nfs-42
- tools-k8s-worker-nfs-43
- tools-k8s-worker-nfs-44
- tools-k8s-worker-nfs-45
- tools-k8s-worker-nfs-46
- tools-k8s-worker-nfs-47
- tools-k8s-worker-nfs-48
- tools-k8s-worker-nfs-5
- tools-k8s-worker-nfs-50
- tools-k8s-worker-nfs-53
- tools-k8s-worker-nfs-54
- tools-k8s-worker-nfs-55
- tools-k8s-worker-nfs-57
- tools-k8s-worker-nfs-58
- tools-k8s-worker-nfs-61
- tools-k8s-worker-nfs-65
- tools-k8s-worker-nfs-66
- tools-k8s-worker-nfs-67
- tools-k8s-worker-nfs-68
- tools-k8s-worker-nfs-69
- tools-k8s-worker-nfs-7
- tools-k8s-worker-nfs-70
- tools-k8s-worker-nfs-71
- tools-k8s-worker-nfs-72
- tools-k8s-worker-nfs-73
- tools-k8s-worker-nfs-74
- tools-k8s-worker-nfs-75
- tools-k8s-worker-nfs-76
- tools-k8s-worker-nfs-77
- tools-k8s-worker-nfs-78
- tools-k8s-worker-nfs-79
- tools-k8s-worker-nfs-8
- tools-k8s-worker-nfs-9
- Upgrade ingress nodes
- tools-k8s-ingress-7
- tools-k8s-ingress-8
- tools-k8s-ingress-9
- Upgrade kubectl on bastions
- Check everything looks good
- Remove the silence in alertmanager
- Revert IRC topic change (only for "tools" cluster)