Page MenuHomePhabricator

VipsScaler clutters /tmp on imagescalers
Closed, ResolvedPublic

Description

Today, /tmp on mw1157 was running out of free space because it was full of huge vips-1*.v files, with size either precisely 400M or slightly less. Also, there's a load of small files called localcopy_5c00a2180a19-1.svg, but they're not influencing usage much. After I nuked all *.v older than 1 day, usage dropped to 40% which means that it will fill everything very soon. We need to investigate why temp files are left to rot forever or at least implement a cleanup cronjob as a stopgap measure.


Version: master
Severity: major

Details

Reference
bz52203

Related Objects

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 1:59 AM
bzimport set Reference to bz52203.
bzimport added a subscriber: Unknown Object (MLST).
MaxSem created this task.Jul 28 2013, 11:11 PM

We could use TempFSFile class, it has a destructor which delete the file on object garbage collection if one asked to do that (with autocollect() ).

It appears vips uses it:

$tmpFile = TempFSFile::factory( 'vips_', $extension );

This is quickly filing up other servers too, all with 400MB, 334M, 337M files so it's becoming a bit urgent:

root@mw1159:/tmp# du -hs
17G	.

The localcopy SVGs are "normal" and preexisting, much older than VipsScaler. In mw1159 it's actually a single file 385 times, base36 6t6jlzs7m7x078w28vct9kyqwg1eacb (also see #49118).

(In reply to comment #3)

This is quickly filing up other servers too, all with 400MB, 334M, 337M files
so it's becoming a bit urgent:

root@mw1159:/tmp# du -hs
17G    .

I suspect (haven't investigated very thouroughly, so may be wrong) what is happening is that the file is created by the vips binary, which is going over a resource limit and getting killed before it can clean up the file.

Perhaps if we add the preconvert option to the VIPS configuration array, it will cause the .v file to be created in a separate command, and then cleaned up by MediaWiki, in a more reliable fashion.

The localcopy SVGs are "normal" and preexisting, much older than VipsScaler.
In
mw1159 it's actually a single file 385 times, base36
6t6jlzs7m7x078w28vct9kyqwg1eacb (also see #49118).

For reference, that's an old version (from 20101013173658) of [[file:Qanat_illustration-de.svg]]

I assume that Ariel's https://gerrit.wikimedia.org/r/#/c/76549/ fixed this server-side.

The patch deletes files matching 'vips-*.v'

We still have to find a way to get MediaWiki to cleanup such files though.

The number of such files created should be significantly reduced now that we use shrink instead of im_shrink

aaron added a comment.Nov 20 2013, 9:29 PM

I don't see any such files anymore and the code uses TempFSFile (and imshrink).

Looks fine on the beta cluster jobrunner as well.

Gilles raised the priority of this task from High to Unbreak Now!.Dec 4 2014, 10:11 AM
Gilles moved this task from Untriaged to Done on the Multimedia board.
Gilles lowered the priority of this task from Unbreak Now! to High.Dec 4 2014, 11:20 AM