Vips scalar hits memory limit when scaling large (100MB) progressive jpeg file
OpenPublic

Description

I'm not sure if this is a bug or not, as there's limited info on what types of images we're actually targeting to fix or not with vips scalar.

https://test2.wikipedia.org/wiki/File:Large-progressive-jpeg.jpg

can't be scaled using Special:VipsTest. If you have bilinear off, it returns error: 'Error creating thumbnail: vips warning: shrink: not integer shrink factors, expect poor results'. With bilinear on, it just does "Error creating thumbnail:".

Given that non-progressive jpegs work fine in our current setup, if vips doesn't work with progressive jpegs, I'm not clear what the benefit is over the current image magick set up for jpeg files.


Version: unspecified
Severity: major

bzimport added a subscriber: wikibugs-l.
bzimport set Reference to bz51275.
Bawolff created this task.Via LegacyJul 13 2013, 3:17 AM
greg added a comment.Via ConduitJul 13 2013, 4:35 AM

Setting to highest/major for the determining if this is really a bug part. I, very unfortunately, am not sure. I don't know all the details.

Jan: can you comment?

bzimport added a comment.Via ConduitJul 15 2013, 8:25 AM

jgerber wrote:

this image is hitting the memory limit.

running
IM_CONCURRENCY=1 vips im_shrink 'Daedongyeojido-full.jpg' test.jpg 2.507838312829525478e+2 2.518363064008394474e+2

without memory limit, it is using about 3Gb here.

greg added a comment.Via ConduitJul 15 2013, 4:59 PM

Ok, thanks.

So, is this something we can heuristically guard against (throw it back to default imagemagick)? What do you think the size limit is for vips on these image types? I see that the image on Commons was thumbnailed correctly: https://commons.wikimedia.org/wiki/File:Daedongyeojido-full.jpg

If you have a good idea about it, please do submit a merge request with the change to the default config settings (per bug 51370).

Bawolff added a comment.Via ConduitJul 15 2013, 5:11 PM

(In reply to comment #3)

Ok, thanks.

So, is this something we can heuristically guard against (throw it back to
default imagemagick)? What do you think the size limit is for vips on these
image types? I see that the image on Commons was thumbnailed correctly:
https://commons.wikimedia.org/wiki/File:Daedongyeojido-full.jpg

If you have a good idea about it, please do submit a merge request with the
change to the default config settings (per bug 51370).

That's a different version of the file. large progressive jpegs are broken on both scalars

greg added a comment.Via ConduitJul 15 2013, 5:14 PM

Ah, my bad. Thanks for the correction.

greg added a comment.Via ConduitJul 15 2013, 5:51 PM

So, everything else in my last comment though stands: what should the limit be and let's get that explicit in Gerrit :)

Aklapper added a comment.Via ConduitAug 15 2013, 3:29 PM

(In reply to comment #6)

So, everything else in my last comment though stands: what should the limit
be and let's get that explicit in Gerrit :)

Jan / bawolff: Any input?

[Pinging as this has been "highest" priority for amonth now.]

Bawolff added a comment.Via ConduitAug 15 2013, 4:29 PM

Reducing priority (its not that big a deal. Progressive jpegs aren't commonly used, and this bug is just the status quo. Enabling vips for tiffs is much more important).

Re greg:
*we don't know if a file is progressive or not at thumbnail time (this just simply would have to be changed in metadata extractor code)
*we would probably want to do more testing on such files to check what performance is like (make sure that we don't just have one outlier). If vips does decompress the whole gile into memory, 50 MP would probably be a good limit. (Our current limit for image magick, if it was enabled for jpeg). However if that's the case there isn't much point using vips for these files.

MarkTraceur added a project: Multimedia.Via WebDec 3 2014, 6:25 PM

Add Comment

Column Prototype
This is a very early prototype of a persistent column. It is not expected to work yet, and leaving it open will activate other new features which will break things. Press "\" (backslash) on your keyboard to close it now.