This is now critical, so I'll add 20GB of space to /.

Can we start a discussion on how to rework these scripts to avoid these kinds of issues?

Mentioned in SAL (#wikimedia-operations) [2025-09-10T09:28:10Z] <claime> cgoubert@deploy1003:/home$ sudo lvextend -L +20G /dev/vg0/root && sudo resize2fs /dev/vg0/root - T404060

I'll shut them down. Working with Tim on what happened here

I have deleted all the ml-* files, some of which were incredibly big. So hopefully that resolves the issue.

I'm running into a problem cancelling the jobs:

foks@deploy1003:~$ kube_env mw-script-restricted-deploy eqiad
bash: kube_env: command not found

I've reached out to Reuven about it.

Got to the bottom of it. All jobs should have been terminated now. There was an issue with the script that was compiling lists with waaaay too many users due to a missed step in the procedure. My apologies for that.

The election was missing the needs-central-list property, so it was trying to mail all users, not just qualified users. All files generated so far should be deleted. The files should be smaller when the script is run again with the property correctly set.

In T404060#11170614, @tstarling wrote:

The election was missing the needs-central-list property, so it was trying to mail all users, not just qualified users. All files generated so far should be deleted. The files should be smaller when the script is run again with the property correctly set.

Following up on this at T404537