Page MenuHomePhabricator

Add support for AVIF: serve AVIF thumbnails where possible
Open, LowestPublic

Description

Piggybacking off of similar WebP bugs such as https://phabricator.wikimedia.org/T152739, AVIF is a royalty-free and open format that has been announced for Blink (Chrome, Edge, Opera) and Gecko (Firefox) rendering engines. For relevant clients, this will result in faster rendering, smaller file transfers, etc. Additionally, support has been added by software vendors (e.g. GIMP) and large tech companies (e.g. Netflix), so it's likely that we will see more AVIF in the wild (which, as far as I'm aware, is still pretty rare for WebP).

https://en.wikipedia.org/wiki/AV1#AV1_Image_File_Format_(AVIF)
https://www.mediawiki.org/wiki/Manual:Adding_support_for_new_filetypes

Event Timeline

@Koavf: "Support" where exactly? Is this only about generating thumbnails as AVIF instead of PNG of images in any kind of format? Is this about uploading files in AVIF format? Please always read and follow https://www.mediawiki.org/wiki/How_to_report_a_bug and be specific where you expect what to happen.

Additionally, support has been added by software vendors (e.g. GIMP)

I don't think so. Citations are needed for such statements.

Generating thumbnails and serving up content in MediaWiki as AVIF images. My goal is to be able to use AVIF images in the exact same way as JPEG and PNG.

Re: GIMP: https://www.zdnet.com/article/chrome-and-firefox-are-getting-support-for-the-new-avif-image-format/

Generating thumbnails and serving up content in MediaWiki as AVIF images. My goal is to be able to use AVIF images in the exact same way as JPEG and PNG.

Creating thumbnails in AVIF format may depend on the backend that you use for creating thumbnails (e.g. ImageMagick, GD, Thumbor, etc). In MediaWiki itself that would be under MediaWiki-File-management (like T27611). On Wikimedia servers that might require support in Thumbor.
Support for/allow uploading files in AVIF format in MediaWiki by default would be a different request under MediaWiki-Uploading.
And allowing uploading files in AVIF format on Wikimedia servers is another, different request under Wikimedia-Site-requests, like T27397.

@Koavf: Please make this task specific which of those is wanted (or maybe something else is wanted), or if more than one thing is wanted then please break it into clear, separate, actionable tasks.
Currently this task is not actionable because it covers way too many things. Thanks a lot!

Koavf renamed this task from Add support for AVIF, serve AVIF thumbnails where possible to Add support for AVIF: serve AVIF thumbnails where possible.Jul 11 2020, 6:22 AM

I suspect this task tree doesn't make much sense in the order its in...

MW will need changes (at minimum) for adding AVIF to the mime map etc. An image handler subclass is probably needed too (especially if we're expecting pretty thumbnails).

And as Andre said, support (in terms of new packages, which can't just necessarily be installed straight off depending on what they are) in the thumbnailers (and the transcoders) may be needed if the current packages (or their installed versions) can't do AVIF

I'm going to mark T257718 and T257719 as stalled, as there's a lot more prep work to be done before either of those can be actioned

It probably also requires a Product type decision of whether "we" want to do this

Adding and maintaing support for an "upcoming" format isn't free. Especially as we don't know if it will recieve enough traction to become viable/widely supported

It does look like TimedMediaHandler has at least some support for https://en.wikipedia.org/wiki/AV1 as part of the WebM handler

The TimedMediaHandler support won't help here directly, except insofar as actually functionally supporting ingestion of AV1 WebM video uploads will require video transcode servers to have libaom installed, so will establish that we can install the codec library on our machines in a sensible way. :)

AVIF will require also a suitable conversion program (perhaps a backported ImageMagick, perhaps something else), and support in both MediaWiki core and thumb0r for the conversion. I've personally found adding support for things to thumb0r to be difficult as it's not easy to set up or run the regression tests.

As noted it's not a zero cost thing, and the bandwidth savings if any would have to be balanced against the disk & memory storage cost of additional file renderings, and the engineering and ops cost of getting it working.

That said it'd be lovely! But we still have serious outstanding issues with our WebP support I'd rather we fix first, especially that we DO NOT SERVE THE RIGHT FILE EXTENSION with WebP thumbnails, causing files to be saved as WebP with a ".png" extension. This really should be fixed before we add more file types into the mix.

I have experience working with Thumbor (including creating test cases), and would be willing to help implement AVIF support if it's decided that it should be a priority. However, ImageMagick only just recently got support for it through libheif 1.7.0 and libaom 2.0.0. Debian doesn't package any version of ImageMagick 7 yet and packages libheif 1.6.1 and libaom 1.0.0 in sid (unstable).

In limited local testing, I found that AVIF encoding is slow. Like, really slow.

$ time convert IMG_4610.jpg IMG_4610.avif

real	0m16.776s
user	0m49.837s
sys	0m1.337s

And that's not even a huge file by Wikimedia standards.

$ identify IMG_4610.avif 
IMG_4610.avif AVIF 4032x3024 4032x3024+0+0 8-bit YCbCr 0.000u 0:00.002

For reference:

$ time convert IMG_4610.jpg IMG_4610.tiff

real	0m0.343s
user	0m0.221s
sys	0m0.097s
$ time convert IMG_4610.jpg IMG_4610.png

real	0m4.600s
user	0m5.002s
sys	0m0.121s

Decoding is on par with other formats though.

$ time convert IMG_4610.avif IMG_4610-2.jpg 

real	0m1.128s
user	0m1.176s
sys	0m0.214s
$ ls -sh1 IMG_4610*
3.7M IMG_4610-2.jpg
2.4M IMG_4610.avif
2.6M IMG_4610.jpg
 17M IMG_4610.png
 35M IMG_4610.tiff

Considering that only 0.2 MB was saved, I don't see the point of using up 85% of the 1min maximum response time to serve images in AVIF. If file creation and editing tools gain widespread support for it, allowing AVIF uploads might be a good idea.

The issue with AVIF is that it's expensive to decode, so much so that on a lot of devices the gains in transfer time are outweighed by the extra expense in decoding time. This tradeoff is the reason why WebP was based on VP8, while VP9 was already available at the time, because VP8 was cheaper to decode. Until hardware decoding of AV1 is widely available on mobile devices, adopting AVIF thumbnails could be counter-productive for a lot of users.

Aklapper triaged this task as Lowest priority.Aug 14 2020, 9:57 AM

Setting priority as per my interpretation of the last comment.

It's probably useful to do some testing on low powered devices and determine the bandwidth at which the faster transfer outweighs the extra decoding time and compare that to the real life internet connectivity data we have. Doing this on a few devices could give us an idea of the percentage of visitors that would currently benefit from such a change.

But deploying this feature blindly would be risky and could result in a worsened experience for a non-negligible portion of visitors.

As it turned out, GIMP added AVIF support to version 2.10.22 in September 2020, see #2668.

This issue is stalling T257718 and T257719.
I believe that what is blocking AVIF support is generating thumbnails from AVIF files, regardless of the thumbnails' format.
I believe that AVIF is well supported by Wikimedia servers' software stack by now.
Whether AVIF is the default format for all thumbnails or whether it saves time with the processing/bandwidth trade-offs is irrelevant and should not be a blocking issue. AVIF support in general is needed for different reasons (12-bit HDR with browser support, namely).

I agree with @Trougnouf that this issue (use AVIF for thumbnails) shouldn't block progress on being able to upload AVIF files.

Using the ImageMagick 6.9.11 implementation of encoding AVIF and HEIF (libavif 0.9.3, libheif 1.12.0) is still painfully slow, using a typical large photograph (extracted PNG from a Canon CR3 raw file):

$ file IMG_0652.png
IMG_0652.png: PNG image data, 6984 x 4660, 8-bit/color RGB, non-interlaced
$ time convert IMG_0652.png IMG_0652.avif
real    0m10.608s
user    0m29.370s
sys     0m2.031s
$ time convert IMG_0652.png IMG_0652.heif
real    0m17.438s
user    0m17.694s
sys     0m0.148s
$ time convert IMG_0652.png IMG_0652.jpg
real    0m0.756s
user    0m0.648s
sys     0m0.092s
This comment was removed by Trougnouf.

I agree with @Trougnouf that this issue (use AVIF for thumbnails) shouldn't block progress on being able to upload AVIF files.

Note that this ticket isn't stalling the other mentioned ticket. Child items/tickets are blocking this ticket and those child items are in the stalling state. I think the work mentioned mostly referred to upgrading thumbor to get the new avif format supported in imagick. This is done, to those items are no longer stalled I think.