The worker nodes in the Kubernetes cluster use the overlay2 storage driver. They therefore use /var/lib/docker/overlay2 for docker storage and such. Right now, that's all on a relatively smallish OS disk. That hasn't caused a huge problem because the disk is like 20G, but we have 60G unused that should be mounted at /var/lib/docker.
The tidiest fixup might be:
- Disable puppet across tools-k8s-worker-*
- Merge a patch that includes something like modules/profile/manifests/ci/dockervolume.pp
- drain the nodes one at a time and do the following to each:
- Stop docker
- Stop kubelet (just in case)
- Delete everything in /var/lib/docker/
- Enable puppet
- Run puppet until bleeding stops (if there is any)
- ensure docker and kubelet are running and happy
- uncordon node