Change Details

for Toolforge see T282942 https://etherpad.wikimedia.org/p/WMCS-2021-11-10-paws-k8s-upgrade == preparation == - disable puppet on all control/worker/ingress nodes [] sudo cumin project:paws # check that it finds the expected hosts [] sudo cumin project:paws 'puppet agent --disable "upgrading k8s rook"' - downtime paws on metricsinfra - https://prometheus-alerts.wmcloud.org [] find a label in https://prometheus.wmcloud.org/cloud/targets that matches project/paws in this case [] select the bell with a \ and enter downtime there. - update project-wide (plus control-specific) version hiera key profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-20' TO profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-21' [] update topic on wikimedia-cloud "Status: Ok" to "Status: upgrading paws k8s" == first api server / paws-k8s-control-1eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-1 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade plan 1.21.8 [] kubeadm upgrade apply 1.21.8 [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-1 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. ** if scheduler permission issue, reboot == paws-k8s-control-2.eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-2 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade node [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-2 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. (scheduler permission issue, rebooted) == paws-k8s-control-3eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-3 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade node [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-3 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. (control manager perm issue, rebooted) run ingress and worker nodes with: user@laptop:~/stuff/wm-puppet.git $ modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control paws-k8s-control-1 --project paws --file nodelist.txt --src 1.20.11 --dst 1.21.8 == ingress workers == (use the script only after following the special procedures in https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes#Ingress_nodes) Though in this case run: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=1 As there are only 2 nodes. Either the first or second will result in some downtime as one needs to kill the remaining pod to run this. Rumor has it that we need to scale like this, though that seems to be rumor, might be better to not scale. paws-k8s-ingress-4 paws-k8s-ingress-3 == workers == paws-k8s-worker-7 paws-k8s-worker-6 paws-k8s-worker-5 paws-k8s-worker-4 paws-k8s-worker-3 paws-k8s-worker-2 paws-k8s-worker-1 === done === === remaining === * based on last time, seems to be safe to run two upgrade processes at once to speed things up ** this was on toolforge, not sure if it applies on paws * NOTE: ingress nodes will get scheduled on normal workers, and need to be deleted to get them running back on the ingress nodes == finishing touches == * revert topic changes on -cloud

for Toolforge see T282942 https://etherpad.wikimedia.org/p/WMCS-2021-11-10-paws-k8s-upgrade == preparation == - disable puppet on all control/worker/ingress nodes [] sudo cumin project:paws # check that it finds the expected hosts [] sudo cumin project:paws 'puppet agent --disable "upgrading k8s rook"' - downtime paws on metricsinfra - https://prometheus-alerts.wmcloud.org [] find a label in https://prometheus.wmcloud.org/cloud/targets that matches project/paws in this case [] select the bell with a \ and enter downtime there. - update project-wide (plus control-specific) version hiera key profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-20' TO profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-21' [] update topic on wikimedia-cloud "Status: Ok" to "Status: upgrading paws k8s" == first api server / paws-k8s-control-1eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-1 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade plan 1.21.8 [] kubeadm upgrade apply 1.21.8 [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-1 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. ** if scheduler permission issue, reboot == paws-k8s-control-2.eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-2 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade node [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-2 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. (scheduler permission issue, rebooted) == paws-k8s-control-3eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-3 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade node [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-3 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. (control manager perm issue, rebooted) run ingress and worker nodes with: user@laptop:~/stuff/wm-puppet.git $ modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control paws-k8s-control-1 --project paws --file nodelist.txt --src 1.20.11 --dst 1.21.8 == ingress workers == (use the script only after following the special procedures in https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes#Ingress_nodes) Though in this case run: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=1 As there are only 2 nodes. Either the first or second will result in some downtime as one needs to kill the remaining pod to run this. Rumor has it that we need to scale like this, though that seems to be rumor, might be better to not scale. paws-k8s-ingress-4 paws-k8s-ingress-3 == workers == paws-k8s-worker-7 paws-k8s-worker-6 paws-k8s-worker-5 paws-k8s-worker-4 paws-k8s-worker-3 paws-k8s-worker-2 paws-k8s-worker-1 === done === === remaining === * based on last time, seems to be safe to run two upgrade processes at once to speed things up ** this was on toolforge, not sure if it applies on paws * NOTE: ingress nodes will get scheduled on normal workers, and need to be deleted to get them running back on the ingress nodes == finishing touches == * merge https://github.com/toolforge/paws/pull/148 * revert topic changes on -cloud

for Toolforge see T282942 https://etherpad.wikimedia.org/p/WMCS-2021-11-10-paws-k8s-upgrade == preparation == - disable puppet on all control/worker/ingress nodes [] sudo cumin project:paws # check that it finds the expected hosts [] sudo cumin project:paws 'puppet agent --disable "upgrading k8s rook"' - downtime paws on metricsinfra - https://prometheus-alerts.wmcloud.org [] find a label in https://prometheus.wmcloud.org/cloud/targets that matches project/paws in this case [] select the bell with a \ and enter downtime there. - update project-wide (plus control-specific) version hiera key profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-20' TO profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-21' [] update topic on wikimedia-cloud "Status: Ok" to "Status: upgrading paws k8s" == first api server / paws-k8s-control-1eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-1 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade plan 1.21.8 [] kubeadm upgrade apply 1.21.8 [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-1 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. ** if scheduler permission issue, reboot == paws-k8s-control-2.eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-2 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade node [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-2 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. (scheduler permission issue, rebooted) == paws-k8s-control-3eqiad1.wikimedia.cloud == [] kubectl drain paws-k8s-control-3 --ignore-daemonsets [] run-puppet-agent --force && apt-get install kubeadm [] kubeadm upgrade node [] apt-get install kubelet kubectl docker-ce containerd.io helm [] systemctl restart kubelet.service docker.service [] cp /etc/kubernetes/admin.conf /root/.kube/config [] kubectl uncordon paws-k8s-control-3 * Don't proceed until you wait a minute or two and observe that the static pods on this control node are NOT crash looping and are working well. (control manager perm issue, rebooted) run ingress and worker nodes with: user@laptop:~/stuff/wm-puppet.git $ modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control paws-k8s-control-1 --project paws --file nodelist.txt --src 1.20.11 --dst 1.21.8 == ingress workers == (use the script only after following the special procedures in https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes#Ingress_nodes) Though in this case run: kubectl -n ingress-nginx-gen2 scale deployment ingress-nginx-gen2-controller --replicas=1 As there are only 2 nodes. Either the first or second will result in some downtime as one needs to kill the remaining pod to run this. Rumor has it that we need to scale like this, though that seems to be rumor, might be better to not scale. paws-k8s-ingress-4 paws-k8s-ingress-3 == workers == paws-k8s-worker-7 paws-k8s-worker-6 paws-k8s-worker-5 paws-k8s-worker-4 paws-k8s-worker-3 paws-k8s-worker-2 paws-k8s-worker-1 === done === === remaining === * based on last time, seems to be safe to run two upgrade processes at once to speed things up ** this was on toolforge, not sure if it applies on paws * NOTE: ingress nodes will get scheduled on normal workers, and need to be deleted to get them running back on the ingress nodes == finishing touches == * merge https://github.com/toolforge/paws/pull/148 * revert topic changes on -cloud