We can have maintain-kubeusers to inject a couple of labels to all resources:

• app.kubernetes.io/managed-by: maintain-kubeusers
• toolforge.org/maintain-kubeusers-git-id: 5bf5e0447b258c3925d248509c5f9c250d2d85d3

Both a missing namespaces or a namespace with the wrong git-id are triggers for maintain-kubeusers to operate.

This means, that when we deploy a new version of maintain-kubeusers, it will loop at least once over all tool account namespaces to:

• query for all namespaced objects that match the first label, but not the second, which is an indication of a resource that needs to be refreshed/recreated.
• add maybe-missing resources (new with this git-id)
• remove maybe-leftover resources (no longer tracked in this git-id)
• refresh the git-id in the namespace resource.

The basic idea sounds good to me. Using the Git hash means that all tools will be processed on the first boot after every maintain-kubeusers commit, which is fine as long as the processing code is written correctly.

There is currently a ConfigMap in each namespace used to track certificate expiration and quota versions already, can we re-use that instead of introducing a new mechanism (namespace label) for tracking per-tool state?

working on a resource abstraction that would allow to store state of each resource in the configmap

Code here: https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23