Since April 2015 Wikimetrics backup is failing and alerting with errors like:
Error: Either failed to get lock on /data/project/wikimetrics/backup/wikimetrics1/hourly, or tar-ing failed.
The reason is that the public reports folder is getting too big: 5k+ folders, 600k+ files, 2.4GB.
The tar-ing of the MySQL and Redis databases takes less than one minute, but the tar-ing of the public reports folder takes more than 1 hour.
This causes the cron jobs to overlap and the consequent file lock issues.
The idea is to compact the old report files so that the tar-ing gets faster. I suppose that we could substitute a report tree:
303903 ├── 2014-12-17 ├── 2014-12-18 ├── 2014-12-19 ... ├── 2015-04-09 ├── 2015-04-10 └── full_report.json
By a compressed file:
This would reduce the size and number of files of the public reports folder.
We could implement this inside the daily_script, so that reports older than N days/months would be compressed.