Mentioned in SAL (#wikimedia-operations) [2016-11-10T00:37:32Z] <godog> remove files on iridium:/tmp older than 5d - T150396

3% more space freed with "apt-get clean"

Also /var/log/account is 800MB, which wouldn't necessarily be an issue though / is very small (10G)

yep, as on einsteinium, process accounting is most of the size of /var/log/, and yes, it should just be extended to larger than 10G

We should do this while reinstalling iridium as phab1001. Ideally we finish phab2001, fail over to that, then reinstall this one with a larger / and the new name.

hmm... I'm not sure what's up with that. Can we just mount it as tmpfs?

Does Phabricator have a config option where to store the temp files?

@Dzahn @mmodell i think it is https://secure.phabricator.com/book/phabricator/article/configuring_file_storage/

"Engine: Local Disk"

Thanks, this sounds like it

storage.local-disk.path: Set to some writable directory on local disk. Make that directory.

We could simply use /srv/tmp because there we have hundreds of gigs while on /t we have like 2. And until this is an issue again we are not going to be on iridium anymore but a reinstalled phab1001 or on phab2001. hmm

Of course there should be some cleanup but we also don't have to make it hard on us by limiting ourselves to this tiny 10G / where you are constantly fighting to keep the remaining 2G free, while we have hundreds free on another mount. Well, we'll change the partman recipe on reinstall, right.

@mmodell tmpfs won't help because it doesn't clean up old files by itself.
@Dzahn @Paladox yeah that might help if it actually gets honoured. My thought on that was that if all phab temp activities are in a single directory it is easier to clean that up recursively instead of just /tmp.

Anyways let's do the bandaid for /tmp now so this doesn't bite us but the issue of course is somewhere in apache/mod_php/phabricator

@fgiunchedi hi, looking at it more it says we should be using MySQL by default so we may have a bug in that since it should not have set a local disk path as I see no setting has been set in puppet.

What about setting a chron to delete those tmp files every day?

Adding the cron with a "find" to delete older files won't be hard, but _how_ old is old enough to delete?

Change 324601 had a related patch set uploaded (by Dzahn):
phab: add cron to clean up old tmp files

https://gerrit.wikimedia.org/r/324601

Change 324601 merged by Dzahn:
phab: add cron to clean up old tmp files

https://gerrit.wikimedia.org/r/324601

Renaming since the issue has been bandaided, what's left to do is investigate and fix the root cause of why phabricator/apache leaves files behind

I see a few places in the phabricator codebase that create temp files but so far I haven't been able to correlate any of it with the files currently seen in /tmp

What I find especially strange is that there seems to be a pattern to the files - most of them are either 4.0K or 8.0K, with occasional file clocking in at 4.1M but I don't see anything other than those three specific file sizes.

Also the 8k files almost all contain a single uppercase A or B. The 4k files are just empty directories.