Support thumbnailing AV1-encoded webm files
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	ToBeFree
	Nov 27 2021, 10:52 AM

Description

List of steps to reproduce (step by step, including full links if applicable):

What happens?:

No thumbnail visible
Error:

Request from [me] via cp3063 cp3063, Varnish XID 937573331
Upstream caches: cp3063 int
Error: 429, Too Many Requests at Sat, 27 Nov 2021 10:48:39 GMT

What should have happened instead?:

Thumbnail visible
.jpg thumbnail file delivered

Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc:
It's an AV1 video, and it's huge, close to the 4GiB limit. However, transcoding has finished successfully for all quality levels after ~2 hours. Just the thumbnail is missing, and the error message ("Too many requests") seems strange to me. At very, very least the error message needs to be improved if AV1 and/or filesize and/or a still-running transcoding process are the issue, as none of this is "Too Many Requests".

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	BUG REPORT	TheDJ	T296562 Support thumbnailing AV1-encoded webm files
Resolved		hnowlan	T216815 Upgrade Thumbor to Buster
Invalid		None	T212941 Investigate systemd hardening to replace Firejail for Thumbor
Resolved		fgiunchedi	T212946 Stream Thumbor logs to logstash
Resolved		• Gilles	T187765 Replace the Nginx fronting Thumbor with Haproxy
Resolved		fgiunchedi	T204266 Backport prometheus haproxy exporter for Jessie
Resolved		jijiki	T220499 Export useful metrics from haproxy logs for Thumbor
Resolved		jijiki	T221562 Build Thumbor packages for buster

Event Timeline

ToBeFree created this task.Nov 27 2021, 10:52 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 27 2021, 10:52 AM

Peachey88 added a project: SRE.Nov 27 2021, 10:58 AM

RhinosF1 added a project: SRE-swift-storage.Nov 27 2021, 11:57 AM

Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptNov 27 2021, 11:57 AM

I'll probably revert the file to the previous revision sometime in the next week, as the filesize is a bit excessive and the codec currently lacks hardware decoding support on client devices. The second link in the bug report then probably needs to be updated (/thumb/archive/... instead of /thumb/...). At the moment, I'm not doing this as I fear it may break the current perfect reproducibility of the bug.

Request from [me] via cp3063 cp3063, Varnish XID 1041209681
Upstream caches: cp3063 int
Error: 429, Too Many Requests at Sat, 27 Nov 2021 13:10:52 GMT

Can someone from SRE confirm that the bug exists and is currently reproducible? Is there a kind of desired "maximum response time" from sysadmins in response to persistent 100% HTTP "client" errors caused by a server-side issue for files embedded on 86 pages including https://commons.wikimedia.org/wiki/Commons:Media_of_the_day ?

AntiCompositeNumber edited projects, added Thumbor; removed SRE-swift-storage, SRE.Nov 30 2021, 11:28 PM

Testing locally:

2021-12-01 00:53:48 thumbor:DEBUG [Video] load_sync: ['/usr/bin/timeout', '--foreground', '59', '/usr/bin/ffprobe', '-v', 'error', '-show_entries', 'format=duration', '-of', 'default=noprint_wrappers=1:nokey=1', 'https://upload.wikimedia.org/wikipedia/commons/4/4a/02-Grey_Squirrel_2021-09-13_nX2.webm']
2021-12-01 00:53:49 thumbor:DEBUG [Video] _parse_time: ['/usr/bin/timeout', '--foreground', '59', '/usr/bin/ffmpeg', '-ss', '87', '-i', 'https://upload.wikimedia.org/wikipedia/commons/4/4a/02-Grey_Squirrel_2021-09-13_nX2.webm', '-y', '-vframes', '1', '-an', '-f', 'image2', '-vf', 'scale=iw*sar:ih', '-nostats', '-loglevel', 'fatal', '/tmp/tmpiObr7a']
2021-12-01 00:53:51 thumbor:DEBUG [Video] _parse_time: ['/usr/bin/timeout', '--foreground', '59', '/usr/bin/ffmpeg', '-ss', '0', '-i', 'https://upload.wikimedia.org/wikipedia/commons/4/4a/02-Grey_Squirrel_2021-09-13_nX2.webm', '-y', '-vframes', '1', '-an', '-f', 'image2', '-vf', 'scale=iw*sar:ih', '-nostats', '-loglevel', 'fatal', '/tmp/tmpzX4QJE']
2021-12-01 00:53:52 thumbor:ERROR [Video] Fprobe/ffmpeg errored
2021-12-01 00:53:52 tornado.access:ERROR 500 GET /thumbor/unsafe/800x/filters:format(jpg)/https://upload.wikimedia.org/wikipedia/commons/4/4a/02-Grey_Squirrel_2021-09-13_nX2.webm (172.17.0.1) 4516.45ms
2021-12-01 00:53:52 thumbor:DEBUG METRICS: timing: response.time:4508
2021-12-01 00:53:52 thumbor:DEBUG METRICS: timing: response.time.500:4508
2021-12-01 00:53:52 thumbor:DEBUG METRICS: inc: response.status.500:1

$ ffprobe 02-Grey_Squirrel_2021-09-13_nX2.webm 
ffprobe version 3.2.16-1+deb9u1 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --prefix=/usr --extra-version=1+deb9u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
[matroska,webm @ 0x55df03c9a680] Unknown/unsupported AVCodecID V_AV1.
[matroska,webm @ 0x55df03c9a680] Could not find codec parameters for stream 0 (Video: none, none(progressive), 1920x1080): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, matroska,webm, from '02-Grey_Squirrel_2021-09-13_nX2.webm':
  Metadata:
    COPYRIGHT       : 
    MAJOR_BRAND     : mp42
    MINOR_VERSION   : 1
    COMPATIBLE_BRANDS: isommp41mp42
    COPYRIGHT-eng   : 
    ENCODER         : Lavf59.8.100
  Duration: 00:02:55.05, start: -0.007000, bitrate: 173856 kb/s
    Stream #0:0: Video: none, none(progressive), 1920x1080, SAR 1:1 DAR 16:9, 60 fps, 60 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      HANDLER_NAME    : Core Media Video
      VENDOR_ID       : [0][0][0][0]
      ENCODER         : Lavc59.12.100 libaom-av1
      DURATION        : 00:02:55.041000000
    Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : Core Media Audio
      VENDOR_ID       : [0][0][0][0]
      ENCODER         : Lavc59.12.100 libopus
      DURATION        : 00:02:55.046000000
Unsupported codec with id 0 for input stream 0

ffmpeg/ffprobe is not built with AV1 support on Debian Stretch. It is supported on Buster, so this is blocked on T216815.

AntiCompositeNumber renamed this task from Persistent "429, Too Many Requests" for Commons AV1 thumbnail to Support thumbnailing AV1-encoded webm files.Dec 1 2021, 1:13 AM

AntiCompositeNumber moved this task from Backlog to Patch merged upstream on the Upstream board.

AntiCompositeNumber triaged this task as Low priority.Dec 1 2021, 1:16 AM

AntiCompositeNumber changed the task status from Open to Stalled.Dec 1 2021, 1:19 AM

Ah well, that explains the issue. I incorrectly assumed that the video converter (AV1 -> VPx) was running on the same host (or at least on the same software) as the thumbnailer. The old VP9 file has been restored for now. Thanks!

TheDJ merged a task: T306301: For all uploaded webm files with AV1 codec no thumb images are generated at all.Apr 22 2022, 2:35 PM

TheDJ added subscribers: C.Suthorn, Winof.

Just a small note: If this issue isn’t going to be fixed soon, I think it might be a good idea to put a warning on the help pages and/or upload tools, so people are made aware that thumbnails are currently broken for AV1 media. That would have spared me some wasted time and headache.

If this helps, I can upload more AV1 media for testing purposes, of various lengths, resolutions, bit-rates etc. Just let me know what you need.

Polda18 awarded a token.Apr 25 2022, 12:45 PM

Polda18 subscribed.

Winof awarded a token.Apr 25 2022, 1:45 PM

Veikk0.ma subscribed.Apr 2 2023, 1:46 PM

Veikk0.ma awarded a token.Apr 2 2023, 1:53 PM

hnowlan closed subtask T216815: Upgrade Thumbor to Buster as Resolved.May 17 2023, 3:47 PM

As I recently found out, popular 8K displays accept 4K media in mp4-h264 format and webm-vp9 and 8k media in mp4-h265 format and webm-av1, but not 8k media in in mp4-h264 format and webm-vp9 and webm-vp8. I uploaded a number of 8k media in webm-vp9 format to commons, but they cannot be shown in native 8k on popular 8k displays. I could reencode this files to av1, but it does not make sense, as the preview images will not work with av1.

C.Suthorn awarded a token.Jul 23 2023, 2:53 PM

Veikk0.ma added a parent task: T209437: Migrate primary Wikimedia video format from VP9 to AV1.Feb 2 2024, 9:36 PM

AntiCompositeNumber removed a parent task: T209437: Migrate primary Wikimedia video format from VP9 to AV1.Feb 2 2024, 11:04 PM

Thumbs do work with AV1 for some time now.

TheDJ closed this task as Resolved.Sat, Jun 15, 1:23 PM

TheDJ claimed this task.