Page MenuHomePhabricator

Create a cron to clean clientbucket every day or hour
Open, LowPublic

Description

Hi, could clientbucket be cleaned regularly with a cron? I found that on a instance in labs that the files in that folder were using a lot of storage. I cleaned like 5gb (estimate) of the files in that folder.

Example:

-r--r----- 1 root 49M Jan 18 18:20 /var/lib/puppet/clientbucket/6/e/2/c/d/4/9/8/6e2cd498423a6d69b20ecbce78c2611e/contents
-r--r----- 1 root 49M Jan 18 18:21 /var/lib/puppet/clientbucket/f/4/5/f/9/a/2/4/f45f9a24a2c91508a4d6823b7d68048d/contents
-r--r----- 1 root 66M Jan 18 18:22 /var/lib/puppet/clientbucket/f/0/2/d/9/d/5/6/f02d9d567a008692c67c5891b657e015/contents
-r--r----- 1 root 49M Jan 18 18:20 /var/lib/puppet/clientbucket/a/7/d/9/0/d/1/0/a7d90d10d1a912e7bd3b5a65b03f3da8/contents
-r--r----- 1 root 29M Jan 18 18:19 /var/lib/puppet/clientbucket/e/b/a/4/a/f/2/4/eba4af2418e8379f3d9a163452d4b163/contents

files were from January. I didn't even realise there was a client bucket folder until today when i tried to clean some storage up.

Event Timeline

Yes, please. My multiple labs instances run out of space in /var and this basically blocks "dpkg". As non-puppet expert, this took me a bit of time to figure out what to do to solve this pb. If I have a look to the solution Mozilla has implemented, this sounds quite trivial.

If I have a look to the solution Mozilla has implemented, this sounds quite trivial.

The option to use "tidy" resource in puppet itself seemed an option but people pointed out it uses way more CPU than a cron/timer with find.

Made the change above to use a systemd timer to delete files older than 14 days but making it an opt-in thing affected users can enable in Hiera. I don't think we can actually know if some are using the client file bucket, though it seems unlikely.

Change 635406 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] base/labs: add systemd timer to clean puppet client bucket

https://gerrit.wikimedia.org/r/635406

@Paladox Please see the change above. still interested in this?

Change 635406 merged by Dzahn:
[operations/puppet@production] base/labs: add systemd timer to clean puppet client bucket

https://gerrit.wikimedia.org/r/635406

dzahn@wikistats-dancing-goat:~$ sudo systemctl start cleanup_puppet_client_bucket.timer
dzahn@wikistats-dancing-goat:~$ sudo systemctl status cleanup_puppet_client_bucket.timer
● cleanup_puppet_client_bucket.timer - Periodic execution of cleanup_puppet_client_bucket.service
   Loaded: loaded (/lib/systemd/system/cleanup_puppet_client_bucket.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Wed 2020-10-28 17:56:21 UTC; 5min ago
  Trigger: Thu 2020-10-29 17:56:24 UTC; 23h left

Oct 28 17:56:21 wikistats-dancing-goat systemd[1]: Started Periodic execution of cleanup_puppet_client_bucket.servic

^ This is the instance I added it to for testing / as an example.

On another instance it was noop of course.

@Paladox Wanna try it?

  • sudo du -hs /var/lib/puppet/clientbucket
  • add "profile::base::labs::cleanup_puppet_client_bucket: true" to your test instance in Hiera
  • run puppet
  • sudo systemctl status cleanup_puppet_client_bucket.timer
  • sudo systemctl start cleanup_puppet_client_bucket.timer
  • sudo systemctl status cleanup_puppet_client_bucket.timer
  • sudo du -hs /var/lib/puppet/clientbucket

?:)

This should be resolved, giving back to original requestor Paladox to confirm it.