Page MenuHomePhabricator

wmr-bot home directory using high NFS storage
Closed, ResolvedPublic

Description

@Wmr-bot, your home directory is the largest one on Toolforge NFS. Since this is a shared environment, anything you can do to clean up unneeded usage will help ensure performance and stability. I identified the following directories as having the most data. Please clean up whatever you can. Thanks!

13G     /home/wmr-bot/01053001-01056000
19G     /home/wmr-bot/pdfdltw2
21G     /home/wmr-bot/pdfdltw
27G     /home/wmr-bot/01029001-01032000

Details

Other Assignee
Wmr-bot

Event Timeline

Dear Bstorm,

They are being moved to cloudstore1008.

Thanks.

wmr-bot

The move is complete.

But why the destiny disappeared?

/mnt/nfs/secondary-cloudstore1008.wikimedia.org-scratch/wmr2
and
/mnt/nfs/secondary-cloudstore1008.wikimedia.org-scratch/wmr3

both gone. Who deleted them?

taavi triaged this task as High priority.Sep 16 2022, 3:19 PM
taavi added a subscriber: taavi.

Hello again. Your user account is again showing up in the list of largest disk space users with roughly 200G of data in its $HOME. What exactly is being stored here and how is it related to the Wikimedia movement?

a bunch of PDFs were added this month, growing the size to 892G:

root@labstore1004:/srv/tools/shared/tools/home/wmr-bot# du -hs tw/pdf
704G    tw/pdf
root@labstore1004:/srv/tools/shared/tools/home/wmr-bot# ls -lh tw/pdf |wc -l
24842
dcaro updated Other Assignee, added: Wmr-bot.

@fnegri @taavi They are for Wikimedia Commons. I will clean up soon.

So, just to summarize:

  • This user account is currently using about 10% of the disk space available to all Toolforge users and tools, which is not acceptable. Please take steps to reduce your disk usage.
  • It's also in violation of Toolforge rules #3 and #4. Please fix those by using a dedicated tool account, and use Kubernetes (preferred) or the job grid instead of running long-running jobs directly on the job grid.
  • The developer account is also missing an email address. Please set one via the Wikitech account preferences. (Note that this email address will be public.)

So, just to summarize:

  • This user account is currently using about 10% of the disk space available to all Toolforge users and tools, which is not acceptable. Please take steps to reduce your disk usage.
  • It's also in violation of Toolforge rules #3 and #4. Please fix those by using a dedicated tool account, and use Kubernetes (preferred) or the job grid instead of running long-running jobs directly on the job grid.
  • The developer account is also missing an email address. Please set one via the Wikitech account preferences. (Note that this email address will be public.)

I will reduce soon. I will need some time before learning how to use the job grid.

Just reduced it from 704G to 641G. I will reduce it gradually in the coming days. I will keep it under 100G after that. The task will be finished in a month, and I will completely remove it after that.

I have added an email address.

Now tw/pdf is under 100G. I shall close this issue.

@wmr thanks for removing those files. I see shanben.ioc.u-tokyo.ac.jp/pdf is still at 173G, could you please clean that up as well? Your home directory is currently at 284G in total, which is by far the largest.