Background
To deploy buildkitd in Kubernetes we use a statefulset with a PersistentVolumeClaim
template for the buildkitd cache volumes. The statefulset is
autoscaled by HPA. When the statefulset is scaled down, the PVCs associated with the
scaled-down pods are not automatically deleted by Kubernetes[1].
Task
Make a program to scan for and delete unbound buildkitd PersistentVolumeClaims that have lingered past a certain amount of time since their last use (suggested default: 4 hours).
PVCs don't contain information about when they're used or who is using them, so the program will need to periodically join the list of running buildkitd pods with the list of PVCs to maintain its own state.
Deploy this program as a Deployment in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner.
Footnotes
[1] We'll be able to do something simpler in the future. There's support for
automatically deleting Statefulset PVCs during scale-down in newer k8s:
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention