(this task is not to be confused with T275555, which asks for Cinder as a replacement for NFS as file storage for individual tools)
|operations/puppet||production||+10 -4||toolforge kubernetes: change class for the new cinder environment|
|Open||None||T286856 Upgrade Toolforge Kubernetes to latest 1.22|
|Open||None||T282942 Upgrade Toolforge Kubernetes to latest 1.21|
|Open||None||T291913 Upgrade PAWS kubernetes to 1.20|
|Resolved||mdipietro||T280402 Upgrade Toolforge Kubernetes to latest 1.20|
|Resolved||Majavah||T280340 Upgrade Toolforge Kubernetes to latest 1.19|
|Resolved||Majavah||T280299 Upgrade Toolforge Kubernetes to latest 1.18|
|Resolved||Majavah||T280302 Upgrade PAWS Kubernetes to the latest 1.18 release|
|Resolved||Bstorm||T246122 Upgrade the Toolforge Kubernetes cluster to v1.16|
|Resolved||Bstorm||T263284 Upgrade Toolforge K8s to 1.17|
|Resolved||Majavah||T264221 Upgrade the nginx ingress controller in Toolforge (and likely PAWS)|
|Resolved||Bstorm||T282087 Support cinder or expanded ephemeral disk worker nodes on Toolforge Kubernetes|
Ingress and control currently use a smaller flavor (g2.cores2.ram4.disk40), we need an equivalent flavor for those. Is 140G ephemeral storage really needed? For example tools-k8s-worker-68 is currently using 40G in /docker, we could use smaller disks to preserve space.
We could, but there isn't much need to. It doesn't actually consume that space on the ceph cluster. This is all thin-provisioned. The docker images we currently use are extremely large (for docker images). I'd rather keep plenty of room.
Control and ingress don't need anywhere near this much space. The control nodes don't have a separate /var/lib/docker mount, and the ingress nodes arguably don't need it either. They don't need more than 2GB of space for docker, and that is fine on the root disk.
I see ingress nodes are using the role::wmcs::toolforge::k8s::worker class in puppet. I can factor out the docker volume and set a different ingress controller class for those that just uses a non-cinder setup. Ingress and control just don't need the extra disk, I don't think.
That patch should make this just fine, and new nodes should be easy to deploy when required after it is merged. When we upgrade the ingress nodes, it'd make sense to set profile::wmcs::kubeadm::docker_vol: false on the appropriate prefix. I don't think that will unmount it retroactively on existing ingress nodes since it is usually hard to make puppet unmount things, right @Andrew ?
Ok updated the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Deploying for ingress and standard worker nodes. The puppet patch is deployed. This should be good to go. We just need to use the correct flavors, etc. No actual cinder storage is required.
the min/max settings are needed for puppet to tell the difference between different volumes (their actual /dev paths are indeterminate). That said, you're free to change the min/max numbers to agree with whatever the sizes are of the flavor you're using.