This alert triggered yesterday:
The NFS drive is over 85% capacity (currently 85.33%) at host paws-nfs-1 in project paws
This is the graph for the past 4 week:
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T429578 PAWS disk space is running out | |||
| Open | Andrew | T431340 Should we ban use of 'freeroot' on PAWS? |
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2026-06-22T15:43:10Z] <andrewbogott> rm -rf /srv/paws/files-to-remove/* for T429578
Comment Actions
There are some very brief docs about this here: https://wikitech.wikimedia.org/wiki/PAWS/Admin#Deleting_user_data_in_case_of_spam_or_credential_leaks
The good news is, those docs are on point for this alert. /srv/paws/files-to-remove had 902 GB of files. I have deleted them!
Comment Actions
followups:
- Add a runbook to the alert -- if it's generic, make an index page for all nfs full alerts
- Add a daily timer to delete all files older than 2 weeks from /srv/paws/files-to-remove/ on the paws NFS server
Comment Actions
Change #1306398 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Add profile::wmcs::daily_file_cleanup
Comment Actions
I'm pretty sure that the cleanup script that copies things into /srv/paws/files-to-remove/ was lost when I rebuilt the NFS server. Going to wait 24 hours to confirm that that's right -- I certainly can't find a job scheduled anyplace.
Comment Actions
Here is a snippet of the kind of thing the attached patch would do. Might be too aggressive...
INFO:__main__:/srv/paws/project/paws/userhomes/6735/botsfight/data/ld_en_wiki.zip has size 2152333312, would move to /srv/paws/files-to-remove/6735/botsfight/data/ld_en_wiki.zip INFO:__main__:/srv/paws/project/paws/userhomes/85256203/swapfile has size 6442450944, would move to /srv/paws/files-to-remove/85256203/swapfile INFO:__main__:/srv/paws/project/paws/userhomes/85256206/root/vms/DarkNodes.img has size 1178861568, would move to /srv/paws/files-to-remove/85256206/root/vms/DarkNodes.img INFO:__main__:/srv/paws/project/paws/userhomes/66059188/enwiki-1-2021.tsv has size 1521694037, would move to /srv/paws/files-to-remove/66059188/enwiki-1-2021.tsv INFO:__main__:/srv/paws/project/paws/userhomes/66059188/enwiki2021.tsv has size 1521694037, would move to /srv/paws/files-to-remove/66059188/enwiki2021.tsv INFO:__main__:/srv/paws/project/paws/userhomes/82817676/freeroot/root/Win7Ult-SP1-SuperLite-x64.iso.gz has size 1050406424, would move to /srv/paws/files-to-remove/82817676/freeroot/root/Win7Ult-SP1-SuperLite-x64.iso.gz INFO:__main__:/srv/paws/project/paws/userhomes/27887995/.cache/huggingface/hub/models--sentence-transformers--stsb-xlm-r-multilingual/blobs/df39fd36487e536df3e2ce8cf2fefa92c4c331cd35098023d765e9ebee057605 has size 1112201288, would move to /srv/paws/files-to-remove/27887995/.cache/huggingface/hub/models--sentence-transformers--stsb-xlm-r-multilingual/blobs/df39fd36487e536df3e2ce8cf2fefa92c4c331cd35098023d765e9ebee057605 INFO:__main__:/srv/paws/project/paws/userhomes/85299283/home/daytona/ubuntu22.qcow2 has size 4063952896, would move to /srv/paws/files-to-remove/85299283/home/daytona/ubuntu22.qcow2 INFO:__main__:/srv/paws/project/paws/userhomes/65401818/freeroot/root/ubuntu.iso has size 2136926208, would move to /srv/paws/files-to-remove/65401818/freeroot/root/ubuntu.iso INFO:__main__:/srv/paws/project/paws/userhomes/64161882/cv_template_pages.tsv has size 2940521246, would move to /srv/paws/files-to-remove/64161882/cv_template_pages.tsv INFO:__main__:/srv/paws/project/paws/userhomes/85352980/0/root/qwen2.5-coder-7b-instruct-q4_k_m.gguf has size 4683073536, would move to /srv/paws/files-to-remove/85352980/0/root/qwen2.5-coder-7b-instruct-q4_k_m.gguf INFO:__main__:/srv/paws/project/paws/userhomes/85352980/0/root/.ollama/models/blobs/sha256-ac9bc7a69dab38da1c790838955f1293420b55ab555ef6b4615efa1c1507b1ed has size 8988110784, would move to /srv/paws/files-to-remove/85352980/0/root/.ollama/models/blobs/sha256-ac9bc7a69dab38da1c790838955f1293420b55ab555ef6b4615efa1c1507b1ed INFO:__main__:/srv/paws/project/paws/userhomes/72593142/sources.db has size 1497751552, would move to /srv/paws/files-to-remove/72593142/sources.db INFO:__main__:/srv/paws/project/paws/userhomes/85306859/freeroot/disk.raw has size 3221225472, would move to /srv/paws/files-to-remove/85306859/freeroot/disk.raw INFO:__main__:/srv/paws/project/paws/userhomes/30575412/hywiki-20260501-pages-articles.xml has size 3827485545, would move to /srv/paws/files-to-remove/30575412/hywiki-20260501-pages-articles.xml INFO:__main__:/srv/paws/project/paws/userhomes/85270248/root/proxmox.iso has size 1706178560, would move to /srv/paws/files-to-remove/85270248/root/proxmox.iso INFO:__main__:/srv/paws/project/paws/userhomes/85270248/root/vms/ubuntu22.img has size 2501640192, would move to /srv/paws/files-to-remove/85270248/root/vms/ubuntu22.img INFO:__main__:/srv/paws/project/paws/userhomes/78232538/media/koume-ena/KSBJ-169.ts.part has size 2179897788, would move to /srv/paws/files-to-remove/78232538/media/koume-ena/KSBJ-169.ts.part INFO:__main__:/srv/paws/project/paws/userhomes/78232538/media/koume-ena/MMB-396.ts has size 3222471152, would move to /srv/paws/files-to-remove/78232538/media/koume-ena/MMB-396.ts INFO:__main__:/srv/paws/project/paws/userhomes/85218311/var/lib/vz/template/iso/ubuntu-22.04.5-live-server-amd64.iso has size 2136926208, would move to /srv/paws/files-to-remove/85218311/var/lib/vz/template/iso/ubuntu-22.04.5-live-server-amd64.iso INFO:__main__:/srv/paws/project/paws/userhomes/85300698/freeroot/root/vms/ubuntu22.img has size 1178796032, would move to /srv/paws/files-to-remove/85300698/freeroot/root/vms/ubuntu22.img INFO:__main__:/srv/paws/project/paws/userhomes/67158682/randomdir/random.junk has size 1148500000, would move to /srv/paws/files-to-remove/67158682/randomdir/random.junk INFO:__main__:/srv/paws/project/paws/userhomes/5005/netherlands-latest.osm.pbf has size 1158206580, would move to /srv/paws/files-to-remove/5005/netherlands-latest.osm.pbf INFO:__main__:/srv/paws/project/paws/userhomes/79109427/freeroot/usr/local/lib/ollama/libggml-cuda.so has size 1269328368, would move to /srv/paws/files-to-remove/79109427/freeroot/usr/local/lib/ollama/libggml-cuda.so INFO:__main__:/srv/paws/project/paws/userhomes/78202731/freeroot/usr/local/lib/ollama/cuda_v12/libggml-cuda.so has size 1123555280, would move to /srv/paws/files-to-remove/78202731/freeroot/usr/local/lib/ollama/cuda_v12/libggml-cuda.so INFO:__main__:/srv/paws/project/paws/userhomes/78280562/freeroot/freeroot/home/paws/gaianet/Llama-3.2-3B-Instruct-Q5_K_M.gguf has size 2322153760, would move to /srv/paws/files-to-remove/78280562/freeroot/freeroot/home/paws/gaianet/Llama-3.2-3B-Instruct-Q5_K_M.gguf INFO:__main__:/srv/paws/project/paws/userhomes/85322592/freeroot/freeroot/root/vms/pablo.img has size 1162280960, would move to /srv/paws/files-to-remove/85322592/freeroot/freeroot/root/vms/pablo.img INFO:__main__:/srv/paws/project/paws/userhomes/85322592/freeroot/root/vms/pablo.img has size 1163853824, would move to /srv/paws/files-to-remove/85322592/freeroot/root/vms/pablo.img INFO:__main__:/srv/paws/project/paws/userhomes/54485784/en-revisions.tsv has size 1176106258, would move to /srv/paws/files-to-remove/54485784/en-revisions.tsv INFO:__main__:/srv/paws/project/paws/userhomes/76950766/data.txt has size 1656020992, would move to /srv/paws/files-to-remove/76950766/data.txt INFO:__main__:/srv/paws/project/paws/userhomes/85313110/freeroot/freeroot/root/vms/prizmarine.img has size 1576271872, would move to /srv/paws/files-to-remove/85313110/freeroot/freeroot/root/vms/prizmarine.img INFO:__main__:/srv/paws/project/paws/userhomes/85313110/freeroot/freeroot/root/vms/ubuntu24.img has size 1655177216, would move to /srv/paws/files-to-remove/85313110/freeroot/freeroot/root/vms/ubuntu24.img
Comment Actions
I'm asking also because I am not seeing documentation on wikitech about what we clean up on a regular basis
Comment Actions
The initial version of this script is pretty basic, although still slightly more automatic than what we had before:
- Scan for giant files (> 1GB), move to trash dir and replace with an explanatory note
- Log large home directories ( > 20GB)
- Delete files in trash dir older than 28 days
In theory the 'with an explanatory note' bit gives users a month to push back if we're being overly aggressive about yanking files. As far I know, no one has ever pushed back.
For the next iteration I might also detect and remove any 'freeroot' directories regardless of size. I'm interested to hear about legit use cases for this though.
Comment Actions
Change #1306398 merged by Andrew Bogott:
[operations/puppet@production] Add profile::wmcs::nfs::daily_file_cleanup
Comment Actions
Change #1307823 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] profile::wmcs::nfs::daily_file_cleanup: fully qualify path to 'find'
Comment Actions
Change #1307823 merged by Andrew Bogott:
[operations/puppet@production] profile::wmcs::nfs::daily_file_cleanup: fully qualify path to 'find'
Comment Actions
A bit of documentation added here: https://wikitech.wikimedia.org/w/index.php?title=PAWS%2FGetting_started_with_PAWS&diff=2433391&oldid=2254420
Comment Actions
the amount of paws disk space available is low and getting worse as far as i can see. what actions are going to make a measurable difference?
Comment Actions
@Andrew the service to delete the files was failing due to permissions, just removed them manually:
root@paws-nfs-2:~# systemctl status daily_file_cleanup.service
× daily_file_cleanup.service - Delete files older than 21 from /srv/paws/files-to-remove
Loaded: loaded (/usr/lib/systemd/system/daily_file_cleanup.service; static)
Active: failed (Result: exit-code) since Sat 2026-07-11 01:00:01 UTC; 11h ago
Invocation: aa2adeeb5d5143d69d1cb54208f552c1
TriggeredBy: ● daily_file_cleanup.timer
Docs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
Process: 827820 ExecStart=/usr/bin/find /srv/paws/files-to-remove -mindepth 1 -mtime +21 -delete (code=exited, status=1/FAILURE)
Main PID: 827820 (code=exited, status=1/FAILURE)
Mem peak: 5.8M
CPU: 43ms
Jul 11 01:00:01 paws-nfs-2 find[827820]: /usr/bin/find: cannot delete ‘/srv/paws/files-to-remove/77069620/.my.cnf’: Operation not permitted
Jul 11 01:00:01 paws-nfs-2 find[827820]: /usr/bin/find: cannot delete ‘/srv/paws/files-to-remove/33556556/.npm’: Directory not empty
Jul 11 01:00:01 paws-nfs-2 find[827820]: /usr/bin/find: cannot delete ‘/srv/paws/files-to-remove/33556556/.local/share’: Directory not empty
Jul 11 01:00:01 paws-nfs-2 find[827820]: /usr/bin/find: cannot delete ‘/srv/paws/files-to-remove/33556556/.local’: Directory not empty
Jul 11 01:00:01 paws-nfs-2 find[827820]: /usr/bin/find: cannot delete ‘/srv/paws/files-to-remove/33556556/.ipython’: Directory not empty
Jul 11 01:00:01 paws-nfs-2 find[827820]: /usr/bin/find: cannot delete ‘/srv/paws/files-to-remove/33556556/.my.cnf’: Operation not permitted
Jul 11 01:00:01 paws-nfs-2 find[827820]: /usr/bin/find: cannot delete ‘/srv/paws/files-to-remove/79243345/.my.cnf’: Operation not permitted
Jul 11 01:00:01 paws-nfs-2 systemd[1]: daily_file_cleanup.service: Main process exited, code=exited, status=1/FAILURE
Jul 11 01:00:01 paws-nfs-2 systemd[1]: daily_file_cleanup.service: Failed with result 'exit-code'.
Jul 11 01:00:01 paws-nfs-2 systemd[1]: Failed to start daily_file_cleanup.service - Delete files older than 21 from /srv/paws/files-to-remove.Comment Actions
Found also errors on previous run of the unit that move the big files around (running not to try to clean a bit more space):
root@paws-nfs-2:~# journalctl -u scan_and_move_large_files.service -f Jul 10 14:54:06 paws-nfs-2 scan-and-shrink[808278]: ^ Jul 10 14:54:06 paws-nfs-2 scan-and-shrink[808278]: File "/usr/local/sbin/scan-and-shrink", line 81, in scan Jul 10 14:54:06 paws-nfs-2 scan-and-shrink[808278]: os.rename(homedir, dest_path) Jul 10 14:54:06 paws-nfs-2 scan-and-shrink[808278]: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ Jul 10 14:54:06 paws-nfs-2 scan-and-shrink[808278]: OSError: [Errno 39] Directory not empty: '/srv/paws/project/paws/userhomes/79243345' -> '/srv/paws/files-to-remove/79243345' Jul 10 14:54:06 paws-nfs-2 systemd[1]: scan_and_move_large_files.service: Main process exited, code=exited, status=1/FAILURE Jul 10 14:54:06 paws-nfs-2 systemd[1]: scan_and_move_large_files.service: Failed with result 'exit-code'. Jul 10 14:54:06 paws-nfs-2 systemd[1]: Failed to start scan_and_move_large_files.service - Moves large files via scan-and-shrink.
Comment Actions
Change #1311071 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] scan-and-shrink: use shutil.move instead of os.rename for directories
Comment Actions
Change #1311071 merged by Andrew Bogott:
[operations/puppet@production] scan-and-shrink: use shutil.move instead of os.rename for directories
Comment Actions
Closing for now as we have cleaned up and also automated the clean up script on T433572