Page MenuHomePhabricator

Allow cleaning up specific mwscript-k8s runs
Closed, DeclinedPublic

Description

After figuring out why the broken mwscript-k8s run in T369142 failed, it was not clear to me how to delete it – mwscript-cleanup --dry-run eqiad printed no output (because the release was too recent). I eventually pieced this command together based on mwscript-cleanup’s source code:

lucaswerkmeister-wmde@deploy1002 ~ $ RELEASE_NAME=r72z2aop helmfile --file /srv/deployment-charts/helmfile.d/services/mw-script/helmfile.yaml --environment eqiad --selector name=r72z2aop destroy
Listing releases matching ^r72z2aop$
r72z2aop	mw-script	1       	2024-07-03 08:40:17.331229287 +0000 UTC	deployed	mediawiki-0.6.35	           

Deleting r72z2aop
release "r72z2aop" uninstalled


DELETED RELEASES:
NAME
r72z2aop

IMHO it would be useful if mwscript-cleanup took an optional list of specific releases to clean up regardless of their age. This would also give me more confidence that I’m not accidentally cleaning up anything someone else might still be interested in.

Event Timeline

RLazarus subscribed.

Thanks! In general you shouldn't need to do this, even if the job was a mistake. Kubernetes cleans up the job automatically a week after it terminates -- whether it completed or failed -- and there's nothing wrong with leaving it until then.

We use mwscript-cleanup as a systemd timer to additionally tidy up Helm secrets (associated with the implementation detail that each job is a new Helm release) but that's all hidden plumbing; I don't think it's what you care about.

If it's important to delete the job ahead of schedule, the best way is probably just a kubectl delete command:

$ kube_env mw-script-deploy eqiad  # Carefully use the -deploy user to get delete privileges
$ kubectl get job -l username=lucaswerkmeister-wmde  # Look up the job name, if you don't have it (mwscript-k8s prints it now)
$ kubectl delete job mw-script.codfw.r72z2aop

That automatically deletes the associated pod as well, terminating it if it's still running. It doesn't do anything with the Helm secret, but mwscript-cleanup still will when it runs later.