Page MenuHomePhabricator

Phabricator leaving old files in /tmp
Open, MediumPublic

Description

Looks like free space on iridium keeps slowly decreasing and it is now at 5% free (it was 32% on Oct 26th).
In this case it is usually /var/log and/or /tmp, it seems there are very old files in the latter. Most files are git diff outputs and in total /tmp is around 1GB (ditto for /var/log at 1.5GB).

root@iridium:/tmp# ls -latr | head -10
total 7520
drwxr-xr-x   22 root     root        4096 Oct 20 14:42 ..
drwx------    2 www-data www-data    4096 Oct 25 20:31 d32twc5br14ccsc8
drwx------    2 www-data www-data    4096 Oct 25 20:31 5qvpp4olvig40w4c

Details

Related Gerrit Patches:
operations/puppet : productionphab: add cron to clean up old tmp files

Event Timeline

Restricted Application added subscribers: TerraCodes, Aklapper. · View Herald TranscriptNov 10 2016, 12:35 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-10T00:37:32Z] <godog> remove files on iridium:/tmp older than 5d - T150396

Dzahn added a subscriber: Dzahn.Nov 10 2016, 12:40 AM

3% more space freed with "apt-get clean"

Also /var/log/account is 800MB, which wouldn't necessarily be an issue though / is very small (10G)

yep, as on einsteinium, process accounting is most of the size of /var/log/, and yes, it should just be extended to larger than 10G

We should do this while reinstalling iridium as phab1001. Ideally we finish phab2001, fail over to that, then reinstall this one with a larger / and the new name.

fgiunchedi triaged this task as Medium priority.Nov 30 2016, 2:07 AM
fgiunchedi added a subscriber: mmodell.

/tmp keeps getting full with temporary directories that are never cleaned up. Interestingly all files in there are either one byte or 4194304 bytes so some kind of limit in action? Anyways it'd be good to have some cleanup in place, @mmodell have you seen anything like that from phab?

hmm... I'm not sure what's up with that. Can we just mount it as tmpfs?

Dzahn added a comment.Nov 30 2016, 6:42 PM

Does Phabricator have a config option where to store the temp files?

Dzahn added a comment.Nov 30 2016, 6:59 PM

Thanks, this sounds like it

storage.local-disk.path: Set to some writable directory on local disk. Make that directory.

We could simply use /srv/tmp because there we have hundreds of gigs while on /t we have like 2. And until this is an issue again we are not going to be on iridium anymore but a reinstalled phab1001 or on phab2001. hmm

Dzahn added a comment.Nov 30 2016, 7:02 PM

Of course there should be some cleanup but we also don't have to make it hard on us by limiting ourselves to this tiny 10G / where you are constantly fighting to keep the remaining 2G free, while we have hundreds free on another mount. Well, we'll change the partman recipe on reinstall, right.

@mmodell tmpfs won't help because it doesn't clean up old files by itself.
@Dzahn @Paladox yeah that might help if it actually gets honoured. My thought on that was that if all phab temp activities are in a single directory it is easier to clean that up recursively instead of just /tmp.

Anyways let's do the bandaid for /tmp now so this doesn't bite us but the issue of course is somewhere in apache/mod_php/phabricator

@fgiunchedi hi, looking at it more it says we should be using MySQL by default so we may have a bug in that since it should not have set a local disk path as I see no setting has been set in puppet.

What about setting a chron to delete those tmp files every day?

Dzahn added a comment.Nov 30 2016, 8:15 PM

Adding the cron with a "find" to delete older files won't be hard, but _how_ old is old enough to delete?

Change 324601 had a related patch set uploaded (by Dzahn):
phab: add cron to clean up old tmp files

https://gerrit.wikimedia.org/r/324601

Change 324601 merged by Dzahn:
phab: add cron to clean up old tmp files

https://gerrit.wikimedia.org/r/324601

fgiunchedi renamed this task from iridium / filesystem almost full to Phabricator leaving old files in /tmp.Dec 1 2016, 8:20 PM

Renaming since the issue has been bandaided, what's left to do is investigate and fix the root cause of why phabricator/apache leaves files behind

I see a few places in the phabricator codebase that create temp files but so far I haven't been able to correlate any of it with the files currently seen in /tmp

What I find especially strange is that there seems to be a pattern to the files - most of them are either 4.0K or 8.0K, with occasional file clocking in at 4.1M but I don't see anything other than those three specific file sizes.

Also the 8k files almost all contain a single uppercase A or B. The 4k files are just empty directories.