Page MenuHomePhabricator

Panoramic images thumbnailing component
Open, MediumPublic

Description

User experience

Suppose you have a large spherical equirectangular image called Pano.jpg. You embed it in a page with syntax such as

[[File:Pano.jpg|400px|thumb|photosphere]]

I am proposing the use of a "photosphere" image parameter which will force the image to be handled as a spherical panorama.

Initially, in the page view, we want to show a thumbnail of the image, similar to what it will look like after click in the pan-and-zoom interface. This requires reprojection from equirectangular to rectilinear, and cropping to a reduced field of view.

Example source imageSimple central crop, no reprojectionReprojection with PTmender
800px-Pano.jpg (400×800 px, 58 KB)
equicrop-400.jpg (400×400 px, 100 KB)
pano-ypr000.jpg (400×400 px, 72 KB)

Parameters

The yaw, pitch, roll, field of view and aspect ratio are all free parameters which need to be chosen somehow. Pannellum extracts yaw, pitch and roll from the GPano metadata if it is available, and uses the image centre by default otherwise. That seems like a reasonable convention to follow.

Pannellum uses a default initial horizontal FOV of 100°, which is very wide by photography standards. The Wikipedia article on angle of view puts it into the "ultra-wide" category. I would welcome feedback on the choice of this parameter.

A default aspect ratio of 4:3 seems reasonable although I would welcome feedback on that also, and on whether there should be a thumbnail parameter to override it.

In the portrait orientation, Pannellum's vertical field of view ends up being so large as to cause distracting distortions. Note that rectilinear projection fails at FOV ≥180°. A bounding box of say 100°x100° is probably more reasonable than an unlimited vertical field of view.

Backend

After reviewing a few alternatives, I am suggesting that we use PTmender from the libpano13-bin package for this. It is poorly documented and the C code is scary, but it is a small binary and it does the job. As long as we control the input filenames and script contents, it probably won't overflow its stack buffers. It's not parsing EXIF, it just reads the image data, so the attack surface is not too bad.

It only supports TIFF output, so it's necessary to pass the resulting image through ImageMagick to convert it to a JPEG.

There will be an implementation in core and an implementation in Thumbor. MediaWiki will include an spc option in the image filename, short for spherical panorama crop. This will distinguish flat thumbnails from reprojected thumbnails.

Event Timeline

It only supports TIFF output, so it's necessary to pass the resulting image through ImageMagick to convert it to a JPEG.

What input formats are supported?

It only supports TIFF output, so it's necessary to pass the resulting image through ImageMagick to convert it to a JPEG.

What input formats are supported?

PPM, JPEG, TIFF, PNG, PSD, and "HDR" which is apparently the RGBE file format by Greg Ward. But in MediaWiki, I don't know if there is a need to support formats other than JPEG. For prototyping purposes, JPEG is definitely the easiest to implement. I know the pros love TIFF, but for performance, it is not ideal.

In https://www.mediawiki.org/wiki/User:TheDJ/panoviewer i have so far not dealt with the issue of rectifying the image preview. It was one of the issues i couldnt really find a good solution for so far. I like this idea.

Pannellum extracts yaw, pitch and roll from the GPano metadata if it is available

In my experiments so far, i've noticed that we also have a LOT of images that do NOT have GPano metadata.
I've played with adding a parserfunction for the file page, to allow you to specify gpano metadata. See https://gitlab.wikimedia.org/hartman/panoviewer/-/blob/main/includes/Hooks.php?ref_type=heads#L99

A parserfunction is probably an easy quickfix for this. long term i'd prefer to have commonsdata statements however.

In my experiments so far, i've noticed that we also have a LOT of images that do NOT have GPano metadata.

Probably less than half have it. We could have a tool that adds it, although it does seem inefficient to duplicate a 100MB image just to change the default thumbnail.

I've played with adding a parserfunction for the file page, to allow you to specify gpano metadata. See https://gitlab.wikimedia.org/hartman/panoviewer/-/blob/main/includes/Hooks.php?ref_type=heads#L99

A parserfunction is probably an easy quickfix for this. long term i'd prefer to have commonsdata statements however.

Thumbor doesn't have database access, so if we want to support this, we will need MediaWiki to send the yaw/pitch/roll parameters in the filename.

An extra complication is that there are two coordinate systems for yaw/pitch/roll -- real world and image relative. Pannellum configuration uses real world coordinates and uses GPano "pose" metadata to convert it to image coordinates. But for thumbnailing with a naïve backend like PTmender we will need to send image coordinates.