Page MenuHomePhabricator

vp9 transcode failures with signal 15
Closed, ResolvedPublic

Description

Getting some VP9 video transcode failures with signal 15

wikiadmin@10.64.48.150(commonswiki)> select transcode_image_name,transcode_key,transcode_error from transcode where transcode_key like '%.vp9.%' and transcode_error != ''\G
*************************** 1. row ***************************
transcode_image_name: (2017.10.12_영화의전당_야외극장)_제22회_BIFF_부산국제영화제_2017_개막식_레드카펫.webm
       transcode_key: 720p.vp9.webm
     transcode_error: '/usr/bin/ffmpeg' -y -i '/tmp/localcopy_c2f7fd24b5ca.webm' -threads 8 -row-mt 1 -crf '32' -qmin '10' -qmax '51' -vb '2560000' -vcodec libvpx-vp9 -tile-columns '2' -auto-alt-ref 1 -lag-in-frames 25 -g '240' -speed '2' -f webm -s 1280x720 -ab 96000 -ar '48000' -acodec 'libopus' -pass '2' -passlogfile '/tmp/transcode_720p.vp9.webm676f5c4bf44e.webm.log' /tmp/transcode_720p.vp9.webm676f5c4bf44e.webm

Exitcode: -1
Memory: 4194304

ffmpeg version 3.2.12-1~deb9u1+wmf1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --prefix=/usr --extra-version='1~deb9u1+wmf1' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, matroska,webm, from '/tmp/localcopy_c2f7fd24b5ca.webm':
  Metadata:
    encoder         : Lavf57.25.100
  Duration: 00:28:16.60, start: 0.000000, bitrate: 4752 kb/s
    Stream #0:0: Video: vp8, yuv420p, 1920x1080, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
[libvpx-vp9 @ 0x5650a178ae00] v1.7.0
Output #0, webm, to '/tmp/transcode_720p.vp9.webm676f5c4bf44e.webm':
  Metadata:
    encoder         : Lavf57.56.101
    Stream #0:0: Video: vp9 (libvpx-vp9), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=10-51, 2560 kb/s, 30 fps, 1k tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.64.101 libvpx-vp9
    Side data:
      cpb: bitrate max/min/avg: 0/0/2560000 buffer size: 0 vbv_delay: -1
Stream mapping:
  Stream #0:0 -> #0:0 (vp8 (native) -> vp9 (libvpx-vp9))
Press [q] to stop, [?] for help
frame=31974 fps= 13 q=0.0 size=  281006kB time=00:17:44.96 bitrate=2161.6kbits/s speed=0.444x    
*************************** 2. row ***************************
transcode_image_name: (2017.10.12_영화의전당_야외극장)_제22회_BIFF_부산국제영화제_2017_개막식_레드카펫.webm
       transcode_key: 1080p.vp9.webm
     transcode_error: '/usr/bin/ffmpeg' -y -i '/tmp/localcopy_6913ce02a40c.webm' -threads 8 -row-mt 1 -crf '31' -qmin '9' -qmax '51' -vb '5120000' -vcodec libvpx-vp9 -tile-columns '4' -auto-alt-ref 1 -lag-in-frames 25 -g '240' -speed '2' -f webm -s 1920x1080 -ab 96000 -ar '48000' -acodec 'libopus' -pass '2' -passlogfile '/tmp/transcode_1080p.vp9.webmebbf02e98085.webm.log' /tmp/transcode_1080p.vp9.webmebbf02e98085.webm

Exitcode: -1
Memory: 4194304

ffmpeg version 3.2.12-1~deb9u1+wmf1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --prefix=/usr --extra-version='1~deb9u1+wmf1' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, matroska,webm, from '/tmp/localcopy_6913ce02a40c.webm':
  Metadata:
    encoder         : Lavf57.25.100
  Duration: 00:28:16.60, start: 0.000000, bitrate: 4752 kb/s
    Stream #0:0: Video: vp8, yuv420p, 1920x1080, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
[libvpx-vp9 @ 0x5592bccfde00] v1.7.0
Output #0, webm, to '/tmp/transcode_1080p.vp9.webmebbf02e98085.webm':
  Metadata:
    encoder         : Lavf57.56.101
    Stream #0:0: Video: vp9 (libvpx-vp9), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=9-51, 5120 kb/s, 30 fps, 1k tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.64.101 libvpx-vp9
    Side data:
      cpb: bitrate max/min/avg: 0/0/5120000 buffer size: 0 vbv_delay: -1
Stream mapping:
  Stream #0:0 -> #0:0 (vp8 (native) -> vp9 (libvpx-vp9))
Press [q] to stop, [?] for help
frame=32677 fps=7.6 q=0.0 Lsize=  543113kB time=00:18:09.20 bitrate=4084.8kbits/s speed=0.252x    
video:542879kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.043139%
Exiting normally, received signal 15.

Event Timeline

Unfortunately the error logs in transcode table don't include the hostname yet.

Can confirm they were all around the same time:

wikiadmin@10.64.48.150(commonswiki)> select transcode_image_name,transcode_key,transcode_time_error from transcode where transcode_key like '%.vp9.%' and transcode_error != '';
+----------------------------------------------------------------------------------------------------------------+----------------+----------------------+
| transcode_image_name                                                                                           | transcode_key  | transcode_time_error |
+----------------------------------------------------------------------------------------------------------------+----------------+----------------------+
| (2017.10.12_영화의전당_야외극장)_제22회_BIFF_부산국제영화제_2017_개막식_레드카펫.webm                          | 720p.vp9.webm  | 20180801051423       |
| (2017.10.12_영화의전당_야외극장)_제22회_BIFF_부산국제영화제_2017_개막식_레드카펫.webm                          | 1080p.vp9.webm | 20180801055542       |
| (3-5)_Waarom_genieten_vrouwen_minder_van_seks_dan_mannen-.webm                                                 | 1080p.vp9.webm | 20180801051439       |
| (3-5)_Waarom_moét_je_soms_wel_naar_een_advertentie,_verkeerslicht_of_bitterbal_kijken-.webm                    | 720p.vp9.webm  | 20180801051646       |
| (3-5)_Waarom_moét_je_soms_wel_naar_een_advertentie,_verkeerslicht_of_bitterbal_kijken-.webm                    | 1080p.vp9.webm | 20180801053300       |
| (3-5)_Waarom_zijn_internationale_oorlogstribunalen_niet_altijd_een_goed_idee-.webm                             | 720p.vp9.webm  | 20180801052031       |
| (3-5)_Wat_zit_er_onder_bijna_elk_vakantie-eiland_verborgen-.webm                                               | 1080p.vp9.webm | 20180801052257       |
| (361VR)_Superwoman_주이형의_비키니몸매_만들기_운동법_대공개!_-_On_VR.webm                                      | 2160p.vp9.webm | 20180801051439       |
| (4-5)_Hoe_kan_zelfs_een_brave_huisvader_op_straat_ineens_gewelddadig_worden-.webm                              | 1080p.vp9.webm | 20180801053303       |
| (4-5)_Hoe_kom_je_erachter_waaraan_iemand_overleden_is-.webm                                                    | 480p.vp9.webm  | 20180801052031       |
| (4-5)_Hoe_kom_je_erachter_waaraan_iemand_overleden_is-.webm                                                    | 1080p.vp9.webm | 20180801055540       |
| (4-5)_Hoe_kun_je_seksueel_misbruik_verwerken-.webm                                                             | 720p.vp9.webm  | 20180801054708       |
| (4-5)_Hoe_werd_ontdekt_dat_er_een_bol_bijna_zo_groot_als_de_maan_in_de_aarde_zit-.webm                         | 720p.vp9.webm  | 20180801055543       |
| (4-5)_Moet_een_arts_je_familie_inlichten_als_jij_je_erfelijke_afwijking_verzwijgt-.webm                        | 480p.vp9.webm  | 20180801054646       |
| (4-5)_Moet_een_arts_je_familie_inlichten_als_jij_je_erfelijke_afwijking_verzwijgt-.webm                        | 720p.vp9.webm  | 20180801055544       |
| (4-5)_Waarom_is_chocolade_volgens_de_wetenschap_het_allerlekkerste_wat_er_is-.webm                             | 480p.vp9.webm  | 20180801054903       |
| (4-5)_Waarom_is_chocolade_volgens_de_wetenschap_het_allerlekkerste_wat_er_is-.webm                             | 720p.vp9.webm  | 20180801055544       |
| (4-5)_Waarom_is_chocolade_volgens_de_wetenschap_het_allerlekkerste_wat_er_is-.webm                             | 1080p.vp9.webm | 20180801054903       |
| (4-5)_Waarom_kan_het_soms_beter_zijn_om_een_dader_niet_te_vervolgen-.webm                                      | 120p.vp9.webm  | 20180801054646       |
| (4K)_180624_켄타_KENTA_갸우뚱-_최근에_배운_한국말_Osaka_오사카_팬미팅.webm                                     | 1080p.vp9.webm | 20180801054903       |
| (5-5)_Hoe_kunnen_nota_bene_je_ouders_je_seksleven_verbeteren-.webm                                             | 240p.vp9.webm  | 20180801055544       |
+----------------------------------------------------------------------------------------------------------------+----------------+----------------------+

All 2018-08-01 05:51-05:54 ish. Nothing's noted in the server admin log around that time.

Per joe in irc this is probably from hhvm config restarts, in which case just re-queuing the failed ones should patch em up. Just need to expect more will come and run follow-ups. (Todo: improve retry logic!)

(Re-running the errored ones...)

@brion: Could you please answer the last comment? Thanks in advance!

brion claimed this task.

[old, not currently occurring, ran old items long ago, closing out]