New nodes kubernetes20[1(89)|2(012)] have been handed over by DC-Ops and need to be setup/added do the cluster.
We should set them up with bullseye + overlayfs (T300744) as extra capacity first and decommission kubernetes200[1-4] (which these are replacements for) after the cluster has been completely migrated.
https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_nodes
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Papaul | T299470 (Need By: TBD) rack/setup/install kubernetes20[19|2(012)] | |||
| Resolved | JMeybohm | T300744 Move kubernetes workers to bullseye and docker to overlayfs | |||
| Resolved | JMeybohm | T302208 setup/install kubernetes20[1(89)|2(012)] | |||
| Resolved | Request | Papaul | T303045 decommission kubernetes200[1-4] |
Change 766588 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] install_server: set new partman recipe for new k8s nodes
Change 766588 merged by Elukey:
[operations/puppet@production] install_server: set new partman recipe for new k8s nodes
Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kubernetes2018.codfw.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kubernetes2018.codfw.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kubernetes2019.codfw.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kubernetes2019.codfw.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kubernetes2020.codfw.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kubernetes2020.codfw.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kubernetes2020.codfw.wmnet with OS bullseye executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kubernetes2021.codfw.wmnet with OS bullseye
Change 766808 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] Enable overlayfs for kubernetes20[18-22]
Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kubernetes2021.codfw.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kubernetes2022.codfw.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kubernetes2022.codfw.wmnet with OS bullseye completed:
Change 766808 merged by Elukey:
[operations/puppet@production] Enable overlayfs for kubernetes20[18-22]
All nodes have Bullseye and the new partition layout for overlay. I have also enabled overlay via puppet, and manually added the systemd.unified_cgroup_hierarchy=0 to grub's settings and rebooted (it is applied by profile::kubernetes::node, so convenient for a pre-existing worker but less for a new one).
The hosts seem ready to be added to the k8s codfw cluster, @JMeybohm lemme know how you want to proceed :)
Change 767465 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] Add kubernetes2018 to wikikube codfw
Change 767468 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/homer/public@master] Add BGP config for kubernetes2018
Change 767465 merged by Elukey:
[operations/puppet@production] Add kubernetes2018 to wikikube codfw
Change 767468 merged by Elukey:
[operations/homer/public@master] Add BGP config for kubernetes2018
Change 767482 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] Add kubernetes20[19-22] to wikikube codfw
Change 767485 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/homer/public@master] Add BGP config for kubernetes20[19-22] in wikikube codfw
Status:
Change 767482 merged by Elukey:
[operations/puppet@production] Add kubernetes20[19-22] to wikikube codfw
The first puppet run seems to always end up in:
Notice: /Stage[main]/Profile::Kubernetes::Node/K8s::Kubeconfig[/etc/kubernetes/kubelet_config]/File[/etc/kubernetes/kubelet_config]/ensure: defined content as '{md5}a17b9fafdfa37dd7305196acdef3119d'
Info: /Stage[main]/Profile::Kubernetes::Node/K8s::Kubeconfig[/etc/kubernetes/kubelet_config]/File[/etc/kubernetes/kubelet_config]: Scheduling refresh of Service[kubelet]
Error: Systemd start for kubelet failed!
journalctl log for kubelet:
-- Journal begins at Mon 2022-02-28 15:03:26 UTC, ends at Thu 2022-03-03 09:27:40 UTC. --
-- No entries --
Error: /Stage[main]/K8s::Kubelet/Service[kubelet]/ensure: change from 'stopped' to 'running' failed: Systemd start for kubelet failed!
journalctl log for kubelet:
-- Journal begins at Mon 2022-02-28 15:03:26 UTC, ends at Thu 2022-03-03 09:27:40 UTC. --
-- No entries --
Notice: /Stage[main]/K8s::Kubelet/Service[kubelet]: Triggered 'refresh' from 3 events
Notice: /Stage[main]/Profile::Kubernetes::Node/Base::Expose_puppet_certs[/etc/kubernetes]/File[/etc/kubernetes/ssl]/ensure: created
Notice: /Stage[main]/Profile::Kubernetes::Node/Base::Expose_puppet_certs[/etc/kubernetes]/File[/etc/kubernetes/ssl/cert.pem]/ensure: defined content as '{md5}d344c0d386beca69a0cc66131557b40c'
Notice: /Stage[main]/Profile::Kubernetes::Node/Base::Expose_puppet_certs[/etc/kubernetes]/File[/etc/kubernetes/ssl/server.key]/ensure: defined content as '{md5}2122f5e303b21582ae3fb8bb7ea0b25b'
Notice: /Stage[main]/Profile::Kubernetes::Node/K8s::Kubeconfig[/etc/kubernetes/kubeproxy_config]/File[/etc/kubernetes/kubeproxy_config]/ensure: defined content as '{md5}bc04bfb1a07d81795aa6169157d33dc7'
Info: /Stage[main]/Profile::Kubernetes::Node/K8s::Kubeconfig[/etc/kubernetes/kubeproxy_config]/File[/etc/kubernetes/kubeproxy_config]: Scheduling refresh of Service[kube-proxy]
Error: Systemd start for kube-proxy failed!
journalctl log for kube-proxy:
-- Journal begins at Mon 2022-02-28 15:03:26 UTC, ends at Thu 2022-03-03 09:27:40 UTC. --
-- No entries --
Error: /Stage[main]/K8s::Proxy/Service[kube-proxy]/ensure: change from 'stopped' to 'running' failed: Systemd start for kube-proxy failed!
journalctl log for kube-proxy:
-- Journal begins at Mon 2022-02-28 15:03:26 UTC, ends at Thu 2022-03-03 09:27:40 UTC. --
-- No entries --
Notice: /Stage[main]/K8s::Proxy/Service[kube-proxy]: Triggered 'refresh' from 3 eventsI can't find logs indicating what's wrong, but a subsequent puppet run fixes it.
Change 767485 merged by Elukey:
[operations/homer/public@master] Add BGP config for kubernetes20[19-22] in wikikube codfw
All new nodes up and running!
I didn't spot anything weird, all bgp sessions seem to be up, pods scheduled on the new nodes (for the moment only calico/istio).
Mentioned in SAL (#wikimedia-operations) [2022-03-03T10:18:56Z] <elukey> kubectl cordon kubernetes200[1-4] to avoid scheduling pods on nodes that will be decommed during the next weeks - T302208
Change 773467 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] Set bullseye + overlayfs for kubernetes1017
Change 773467 merged by Elukey:
[operations/puppet@production] Set bullseye + overlayfs for kubernetes1017