Use VipsScaler to scale TIFFs
OpenPublic

Description

14:58 < bawolff> greg-g: The most likely change I imagine people will want is it to start working for tiffs
14:58 < bawolff> greg-g: Which I believe needs a mild amount of dev work to happen
14:58 < bawolff> greg-g: Which I believe needs a mild amount of dev work to happen


Version: unspecified
Severity: enhancement
See Also:
T51118: Limit number of image scaling attempts
T47212: TIFs appear less focused than JPG equivalent

bzimport added a subscriber: wikibugs-l.
bzimport set Reference to bz52045.
greg created this task.Via LegacyJul 25 2013, 7:02 PM
bzimport added a comment.Via ConduitJul 25 2013, 9:51 PM

Bryan.TongMinh wrote:

If I recall correctly, TIFFs are handled by the PagedTiffHandler extension, which should have some basic support for Vips.

gerritbot added a comment.Via ConduitSep 29 2013, 4:36 AM

Change 86413 had a related patch set uploaded by Brian Wolff:
Make PagedTiffHandler work with VipsScaler.

https://gerrit.wikimedia.org/r/86413

gerritbot added a comment.Via ConduitSep 29 2013, 6:32 AM

Change 86416 had a related patch set uploaded by Brian Wolff:
[WIP] Add support for page numbers to VipsScaler.

https://gerrit.wikimedia.org/r/86416

MarkAHershberger added a comment.Via ConduitOct 5 2013, 11:52 PM
  • Bug 25647 has been marked as a duplicate of this bug. ***
Kelson added a comment.Via ConduitMar 11 2014, 9:44 AM

IMO, this is a feature request which looks more and more like a bug report!

Here are the first historical picture uploaded by the Swiss National Library (WiR program):

Not really what I would call "an encouraging start"...

Would be really great is someone could take a few minutes to review this almost 5 months old patch.

Nemo_bis added a comment.Via ConduitMar 26 2014, 6:15 PM

7.36: "Support many additional TIFF variants". http://osdir.com/ml/fedora-package-announce/2013-10/msg01266.html

Aklapper added a comment.Via ConduitApr 17 2014, 10:23 AM

https://gerrit.wikimedia.org/r/#/c/86413/ is still awaiting patch review by somebody from the Multimedia team. See comment 6 why this is important for GLAM.

Bawolff added a comment.Via ConduitApr 17 2014, 4:19 PM

(In reply to Andre Klapper from comment #8)

https://gerrit.wikimedia.org/r/#/c/86413/ is still awaiting patch review by
somebody from the Multimedia team. See comment 6 why this is important for
GLAM.

Its actually waiting on me to finish it (note the [wip] tag). Which I hope to do real soon now(tm)

Fae added a comment.Via ConduitApr 25 2014, 1:01 PM

With regard to the related story (https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/482), as a work-around I am going to try doing a limited run of the images from NYPL but using just one thread. My "problematic" run was using at least 10 threads in parallel. This seems a pragmatic approach if the operational load issue was primarily the quantity of files failing to render per minute.

Should I be able to use the GWToolset this way, it may be possible to reduce the likelyhood of any future incidents by making the use of low numbers of threads when uploading "very large" tiffs a good practice for all users with the GWToolset right.

Bawolff added a comment.Via ConduitApr 26 2014, 8:40 PM

(In reply to Fæ from comment #10)

With regard to the related story
(https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/482),
as a work-around I am going to try doing a limited run of the images from
NYPL but using just one thread. My "problematic" run was using at least 10
threads in parallel. This seems a pragmatic approach if the operational load
issue was primarily the quantity of files failing to render per minute.

Should I be able to use the GWToolset this way, it may be possible to reduce
the likelyhood of any future incidents by making the use of low numbers of
threads when uploading "very large" tiffs a good practice for all users with
the GWToolset right.

This bug isnt really about the gwtoolset issue.

Bawolff added a comment.Via ConduitMay 25 2014, 1:03 AM

(In reply to Bawolff (Brian Wolff) from comment #9)

(In reply to Andre Klapper from comment #8)
> https://gerrit.wikimedia.org/r/#/c/86413/ is still awaiting patch review by
> somebody from the Multimedia team. See comment 6 why this is important for
> GLAM.

Its actually waiting on me to finish it (note the [wip] tag). Which I hope
to do real soon now(tm)

Ok, now it needs reviewers.

gerritbot added a comment.Via ConduitMay 25 2014, 4:42 AM

Change 135289 had a related patch set uploaded by Brian Wolff:
Make tiff thumbnail in 2 steps.

https://gerrit.wikimedia.org/r/135289

gerritbot added a comment.Via ConduitMay 27 2014, 9:40 AM

Change 86416 merged by jenkins-bot:
Add support for page numbers to VipsScaler.

https://gerrit.wikimedia.org/r/86416

gerritbot added a comment.Via ConduitSep 17 2014, 6:04 PM

Change 86413 merged by jenkins-bot:
Make PagedTiffHandler extend TransformationalImageHandler

https://gerrit.wikimedia.org/r/86413

gerritbot added a comment.Via ConduitSep 24 2014, 1:14 PM

Change 135289 merged by jenkins-bot:
Make tiff thumbnail in 2 steps.

https://gerrit.wikimedia.org/r/135289

gerritbot added a comment.Via ConduitOct 2 2014, 9:57 PM

Change 164476 had a related patch set uploaded by Brian Wolff:
Experimentally enable vips for larger (>50MP) tiff files

https://gerrit.wikimedia.org/r/164476

Bawolff added a comment.Via ConduitOct 2 2014, 10:11 PM

I started a thread about this on multimedia mailing list: https://lists.wikimedia.org/pipermail/multimedia/2014-October/000861.html

gerritbot added a comment.Via ConduitOct 2 2014, 10:52 PM

Change 164476 merged by jenkins-bot:
Experimentally enable vips for larger (>50MP) tiff files

https://gerrit.wikimedia.org/r/164476

Bawolff added a comment.Via ConduitOct 3 2014, 1:53 AM

So this is now live on commons experimentally.

There are some really large files (>350 megapixel) that still appear to time out.

It gives broken results for [[File:Amelia_earhart_received_by_president_coolidge.tif]] and [[file:Carr wilbur j honorable.tif]]. This appears to have something to do with how colour spaces are interpreted in grey scale tiffs (Still kind of unclear). It also seems to only appear after sharpening the images. Still not sure what the deal is here.

Bawolff added a comment.Via ConduitOct 3 2014, 2:45 AM

Experimentation locally suggests that running things through

vips im_clip2fmt tempInput.v finalOutput.png 2

may fix the issue mentioned in comment 2

(For reference, the 2 means convert to IM_BANDFMT_USHORT. Previously the convolution had converted it to IM_BANDFMT_FLOAT, which seems to sometimes cause problems when outputting as png. I'm unclear as to the why.)

Bawolff added a comment.Via ConduitOct 3 2014, 3:19 AM

(In reply to Bawolff (Brian Wolff) from comment #21)

Experimentation locally suggests that running things through

vips im_clip2fmt tempInput.v finalOutput.png 2

may fix the issue mentioned in comment 2

(For reference, the 2 means convert to IM_BANDFMT_USHORT. Previously the
convolution had converted it to IM_BANDFMT_FLOAT, which seems to sometimes
cause problems when outputting as png. I'm unclear as to the why.)

Except that breaks other images.

What appears to be happening (I think, behaviour locally and on cluster is kind of different, but related), is that image gets read into vips, using a form of either 8bit integer samples, or 16bit integer sample. Then im_convf converts it to floats to do sharpening, and then when output, it gets converted back to integer samples, but incorrectly.

What I think we need to do:

run vips im_header_int format temp.v before the convolution, save the result, then use im_clip2fmt to convert it back after the convolution.

Kelson added a comment.Via ConduitOct 3 2014, 10:20 AM

I can confirm that all huge TIFF pictures which are in:
https://commons.wikimedia.org/wiki/Category:Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29

... are now *all* thumbnailed correctly.

This is a really important fix because this bug was really weaking us in our GLAM activities in Switzerland. I hope you can keep this patch in production and thank you very much to all people involved in this fix.

@Bawolff, hope to get the chance to meet you at next european hackathon. I definitly owe you a beer.

Aklapper added a comment.Via ConduitNov 12 2014, 1:44 AM

No Gerrit patches left to review here; resetting status.
What's left to do is described in comment 22.

MarkTraceur added a project: Multimedia.Via WebDec 3 2014, 6:25 PM
greg removed a subscriber: greg.Via WebDec 8 2014, 6:08 PM
Nemo_bis awarded a token.Via WebDec 12 2014, 8:01 AM
Kelson added a comment.Via WebJan 7 2015, 11:19 AM

New problems detected with a few big TIFF (but not the biggest), but not sure where/what is the bug.

For example here:
https://commons.wikimedia.org/wiki/File:CH-NB_-_1802,_Zweite_Beschiessung_von_Z%C3%BCrich,_12._und_13._September_-_Collection_Gugelmann_-_GS-GUGE-KELLER-H-E-7.tif

But we have ~50 pictures of this category which are not thumbnailed at all:
https://commons.wikimedia.org/wiki/Category:CH-NB-Collection_Gugelmann

TheDJ added a comment.Via WebJan 7 2015, 11:56 AM

Error creating thumbnail: convert: /tmp/localcopy_7570ffa9a5d6-1.tif: wrong data type 7 for "RichTIFFIPTC"; tag ignored. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/706.
convert: /tmp/localcopy_7570ffa9a5d6-1.tif: Bogus "StripByteCounts" field, ignoring and calculating from imagelength. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/706.
convert: /tmp/localcopy_7570ffa9a5d6-1.tif: Read error at scanline 2357; got 11676 bytes, expected 21900. `TIFFFillStrip' @ error/tiff.c/TIFFErrors/496.

@Kelson this seems unrelated (and likely a problem with the source file). But please open a separate ticket for it if you think it is warranted.

Kelson added a comment.Via WebJan 16 2015, 6:47 PM

@TheDJ Thank you for your analysis. I have open a new ticket https://phabricator.wikimedia.org/T87042

Lokal_Profil added a subscriber: Lokal_Profil.Via WebJan 20 2015, 4:33 PM
Aklapper edited the task description. (Show Details)Via WebMar 9 2015, 2:26 PM
Aklapper set Security to None.

Add Comment

Column Prototype
This is a very early prototype of a persistent column. It is not expected to work yet, and leaving it open will activate other new features which will break things. Press "\" (backslash) on your keyboard to close it now.