clicked "bash" from launcher, got error that seemed related but I don't remember exact wording. eventually after clicking bash a few times got a shell and df in that bash shows /home/paws is full.
Description
Description
Event Timeline
Comment Actions
Just did a quick remove of 'files-to-remove', that freed a little bit of space, enough to let some time for someone to cleanup in depth tomorrow/later
Comment Actions
I freed up a small amount of space. The docs say:
There is a job run to clean user accounts down to 5gb once a day. Files removed from user directories are placed in paws-nfs-1.paws.eqiad1.wikimedia.cloud:/srv/paws/files-to-remove In case a user requests them back. As of writing this has never happened. When the NFS host gets full remove the files in this directory.
That seems to not be happening right now; so far I haven't found where/what that job is.
Comment Actions
If it is running, or is revived, we should probably move from 5G to 4, or expand the nfs volume.
Comment Actions
The abusive accounts had a tendency to install freeroot which has the unfortunate effect of pulling down gigabytes of small files. The shrink script shrinks by removing the largest file one at a time until the home directory is less than 5G, which in this case is about exactly 5G. So reducing to 4G probably won't buy much time. I might recommend looking at all the accounts in the paws home directory created in the last six months or so, and are larger than 1G. If history is any guide the vast majority will be freeroot, which can probably be removed.
Comment Actions
The cleanup cron job was running, but crashed out when it encountered a file it didn't have permission to move. I added some very naive exception handling and re-ran it; it moved 310G of files into /srv/paws/files-to-remove. I also added a bit more verbiage about the script on wikitech, although we should probably get it in git or puppet or something.
In theory there should be a grace period before we delete those files but I will probably do that later today.
Also we need to purge abuser accounts, but that will be less urgent after that 310G is deleted.
Comment Actions
I now have lists of large home directories (more than 1G usage total) that have no date stamps after 2021. Is there any reason to not just delete all of those?
Comment Actions
This is happening again:
@PAWS:~$ df -h | sed -n "1p;4p" Filesystem Size Used Avail Use% Mounted on paws-nfs.svc.paws.eqiad1.wikimedia.cloud:/srv/paws/project/paws/userhomes/34722510 2.9T 2.8T 0 100% /home/paws @PAWS:~$
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2025-12-24T17:21:24Z] <andrewbogott> deleting everything in /srv/paws/files-to-remove T396051
Comment Actions
We have ongoing issues with malicious users gobbling diskspace. I've cleared things up and we're below 50% now, but we may need a better solution for this (maybe starting with just deleting files rather than stashing them in files-to-remove)