Page MenuHomePhabricator

liangent-php is using 348G on Tools
Closed, ResolvedPublic

Description

The majority of this is in two directories:

75G sql
268G mw-log

mw-log in particular is a bit of a mess. It has 2247765 files in it that all seem to be some kind of historical activity log. That is 2+ million log files in a single directory on NFS.

sql has the majority of content in dumps at 75G (just this directory would put this tool in the top 15 of all storage users)

75G dumps

This consists of 920 directories with .sql files for several wiki's going back to 2013:

93M 20131019093307
98M 20131020000022

project/liangent-php/sql# ls -al dumps/20160525000017
total 78024
drwxr-sr-x   2 51117 51117     4096 May 25 00:03 .
drwxr-sr-x 917 51117 51117    36864 May 25 00:00 ..
-rw-r--r--   1 51117 51117  7321424 May 25 00:00 arwiki.sql
-rw-r--r--   1 51117 51117   692300 May 25 00:01 commonswiki.sql
-rw-r--r--   1 51117 51117  1654262 May 25 00:01 enwiki.sql
-rw-r--r--   1 51117 51117   713699 May 25 00:02 mediawikiwiki.sql
-rw-r--r--   1 51117 51117    94141 May 25 00:02 testwikidatawiki.sql
-rw-r--r--   1 51117 51117  1430897 May 25 00:02 wikidatawiki.sql
-rw-r--r--   1 51117 51117    58744 May 25 00:03 zhwikisource.sql
-rw-r--r--   1 51117 51117 67862705 May 25 00:03 zhwiki.sql

There are also 153 error and out files that go back to 2013.

Can we clean out mw-log and what can we do to prevent this kind of growth in the future?

Can we clean out sql/dumps (especially for content that is now years old)?

What is the data retention policy for this tool?

Event Timeline

mw-log is really activity log, for all activities in my php bot, which is actually $wgDebugLogFile. There was a time people asked me "why my bot is doing (something) in (some way)" and the situation was difficult to reproduce. The only use of those files (until now) is to answer those questions for people (and sometimes for myself).

sql/dumps contains database dumps of my user databases on replica servers. Since that service is declared as "unreliable" I have to make my own backups. Previous uses were like T126946.

I‘m running these currently

tools.liangent-php@tools-bastion-03:~/mw-log$ find . -mtime +90 -delete
tools.liangent-php@tools-bastion-03:~/sql/dumps$ rm -rf 2013* & rm -rf 2014* & rm -rf 2015*

but some long-term solution is needed.

@liangent I was looking at old bugs today and ran across this one. Your tool's total disk usage is smaller than before (61G total), but still quite large. 56G of the files are in your $HOME/sql/dumps directory. These files are sql dumps of various projects timestamped from 2016-01-01 to 2017-12-13.

Would you be willing to clean up those files now too?

@liangent Looking closer at your liangent-php tool, I do not see strong signs that it has been in use since 2019-03-25. I have added T253818: Clean up NFS disk hogs that are abandoned as a parent task here to track this apparent abandonment.

Can you please respond as to whether you are still using this tool or not?


Based on the info from T136208#2327065 I have cleaned up old backups in the $HOME/sql/dumps directory. I deleted all of them from prior to 2017-12-01 and compressed the remaining files. This had reduced the total disk usage for your tool from the 61G value mentioned in T136208#6149197 to 5.3G.

If you start using the tool again, I would strongly suggest that you compress your database dumps at the time they are created and also add a pruning job that limits your total backup count to some reasonable number like 7 or 14 total backups at any given time.

Aklapper added a subscriber: liangent.

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

bd808 claimed this task.

Per clean up documented in T136208#6178387