Page MenuHomePhabricator

Panoramic images tiling component
Open, LowPublic

Description

It would be nice to allow users to deeply zoom into large spherical panoramic images, uncovering more detail as they pan and zoom.

We have some large images. In Category:360° panoramas, 9256 images have the correct 2:1 aspect ratio, and of those, 815 images exceed 100 Mpx in size. Attempting to load such a large image on the client side would likely fail due to memory or GPU texture size constraints.

To generate Pannellum's multi-resolution format, we need to reproject the equirectangular source image to a set of 6 cube faces. The cube faces are tiled with a configurable tile size and no overlap. Then the cube faces are reduced in size by a factor of 2 and the tiling process repeats. This continues until the cube faces are a single tile each. There is a python script to do this using Hugin and PIL.

We can either queue a job on upload, like video transcode, or generate tiles on the fly in Thumbor as they are requested. Generating on the fly would be convenient, but the feasibility depends on performance.

With a large test image on my laptop, generating and tiling a single cube face:

$ printf 'o f4 v360 y0 p0 r0\np f0 w10186 h10186 v90 nTIFF_m\n' > ptscript-big
$ /usr/bin/time PTmender  ptscript-big milky.jpg
...
41.74user 0.80system 0:42.57elapsed 99%CPU (0avgtext+0avgdata 2005112maxresident)k
$ /usr/bin/time vips dzsave pano0000.tif mydz --tile-size=1024 --overlap=0 --depth=onetile --vips-concurrency=1
...
2.00user 0.48system 0:01.69elapsed 146%CPU (0avgtext+0avgdata 149012maxresident)k

Maybe it's scraping in under the threshold of feasibility. Investigations will continue.

Event Timeline

Where would we store the results ? Make a new file storage subspace like score-render ?

Generating on the fly would be convenient, but the feasibility depends on performance.

Panellum falls back to a low resolution full size image if the results are not yet ready. The toolforge tool also renders on the fly i believe and it will 'fill in' results as they arrive based on a JS poll.

Where would we store the results ? Make a new file storage subspace like score-render ?

If tiles are generated by Thumbor on the fly then they would be stored as thumbnails. If they are generated by a job, they would go in a separate container like what we do for score.

Panellum falls back to a low resolution full size image if the results are not yet ready. The toolforge tool also renders on the fly i believe and it will 'fill in' results as they arrive based on a JS poll.

There's no Pannellum support for that, it's only in the panoviewer tool. Maybe you're thinking of the fallbackPath parameter, which gives a set of small cube faces to display with the CSS renderer if WebGL fails. The tool polls NFS for the job status and resets Pannellum when the job completes. Maybe that would be feasible in production, but ideally it would just be fast. There's no need for progress feedback if it always just works at high resolution.

By specifying bilinear interpolation I was able to reduce the time taken by PTmender to extract a cube face from 42 to 33 seconds. By default it uses a cubic polynomial on a 4x4 region of the input image, but I think that's excessive given we're only upscaling by a factor of between 1 and 2.

I tried using Hugin to extract a cube face. With one core (OMP_NUM_THREADS=1) it took 41 seconds. Without that environment variable, it used up to 12 cores and took 67 seconds -- its parallel mode was actually slower. The output was visually identical to that produced by PTmender.

I am very tempted to write my own cube face extractor.

DENIVIP's panorama converter shows how simple it can be. It does all 6 cube faces in 67 seconds, single threaded. They messed up the interpolation somehow, the CImg library they used is weird and scary, and the dependencies are wrong (it doesn't really need xorg-dev), but it's certainly inspiring. It's basically doing the thing we need in 229 lines of code.

I did a rough prototype of this in C++, and I got it down to 5.9 seconds for single-threaded extraction of a single cube face. So I think this is the way to go. I'll write a custom cube face extractor, and Thumbor will run it on demand, no need for a job.

Panellum falls back to a low resolution full size image if the results are not yet ready. The toolforge tool also renders on the fly i believe and it will 'fill in' results as they arrive based on a JS poll.

There's no Pannellum support for that, it's only in the panoviewer tool.

Sorry, there is actually an undocumented feature in Pannellum to do this. The JSON configuration object created by generate.py can contain a small base-64 encoded thumbnail.

Jdlrobson moved this task from Backlog to Triaged on the MediaViewer board.