Page MenuHomePhabricator

templatetiger is using 613G in Tools out of 8T
Closed, ResolvedPublic

Description

613G templatetiger

Almost all of this in in templatetiger/public_html/dumps at 612G

Is there some way we can clean this up as a one time thing, is there ongoing cleanup here? I see nothing from 2015 but iirc we may have done that previously for adhoc cleanup.

What is the retention thought process on this tool and can we reduce the storage costs?

Event Timeline

chasemp created this task.May 25 2016, 2:11 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 25 2016, 2:11 PM

It holds also 165G on toolsdb.

templatetiger should be after cleanup now at under 140GB on file system.

templatetiger should be after cleanup now at under 140GB on file system.

What is your cleanup strategy? Files older than n days?

templatetiger should be after cleanup now at under 140GB on file system.

Is there something we can do to prevent this from happening? We don't have a good canonical strategy at the moment but even a cron cleanup monthly is better than where we are now.

Thanks!

chasemp triaged this task as Medium priority.May 31 2016, 3:42 PM

@Kolossos Hi! templatetiger is back up to using 594G - this is a big chunk of our available disk space. Could you please

  • Clean up old data
  • Let us know your cleanup strategy so we can work with you to put in some measures to clean this up regularly and guards against this happening again?

I reduced the volume to the minimum of 150 GB by deleting old files manually.
With the often broken dump files I have no better strategy.

As I still have the problem with sorting big files:
https://phabricator.wikimedia.org/T124822
and I have a problem to restart the webserver of the project, I will discuss with others (like @Bgwhite) if we will end the project.

@yuvipanda : Do you know why templatetiger doesn't work and why I can't bring it back by restarting webserver?
I get an "502 Bad Gateway" . Thanks

BTW: How useful do you think this tool still is?

The error.log file was owned by root, probaly as a result of it having grown super large in the past and someone truncating it. I've removed that now, and the webservice seems back up.

Hi @Kolossos, templatetiger is currently using 868G/8T of tools NFS storage. This is pretty significant usage, could you consider cleaning the directory up?

I removed any of 2016's CheckWiki output files. I kept 2017's files. It should be much lower now.

I update the db to 2017. Everything works fine.
Now the database seems to be deleted/disappear. Anyone any idea?

@Kolossos it's part of the db maintenance happening just now (announced at https://lists.wikimedia.org/pipermail/labs-announce/2017-February/000204.html). Your db was too big to be replicated, so it isn't visible in the host that's online right now. It should be available again in a few hours once maint completes.

@Kolossos it should be back now?

@yuvipanda Yes it works fine.

@Kolossos I noticed that this is back up to 415GB now, we are at pretty high tools NFS usage, it would be great if you could consider cleaning up some of this. Thank you!

Cleaned up to 95 GB.

madhuvishy closed this task as Resolved.Mar 13 2018, 4:39 PM