Upload of MPEG-2 videos was enabled in February 2020 on WMF cluster. Since that moment, no thumbnails were created for those uploads.
Example videos (all are involved):
Specific example:
https://commons.wikimedia.org/wiki/File:Relaisraum_im_Stellwerk_K%C3%B6ln_Kf.mpg
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Add tests for MPEG-1 and MPEG-2 thumbnailing | operations/software/thumbor-plugins | master | +18 -18 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | TheDJ | T166024 Once it's not patent-encumbered, enable MPEG-2 support for Commons uploads and TMH/etc. playback | |||
| Resolved | BUG REPORT | • brooke | T244570 Fix missing thumbnail for MPEG (.mpg) videos | ||
| Resolved | hnowlan | T216815 Upgrade Thumbor to Buster | |||
| Invalid | None | T212941 Investigate systemd hardening to replace Firejail for Thumbor | |||
| Resolved | fgiunchedi | T212946 Stream Thumbor logs to logstash | |||
| Resolved | • Gilles | T187765 Replace the Nginx fronting Thumbor with Haproxy | |||
| Resolved | fgiunchedi | T204266 Backport prometheus haproxy exporter for Jessie | |||
| Resolved | jijiki | T220499 Export useful metrics from haproxy logs for Thumbor | |||
| Resolved | jijiki | T221562 Build Thumbor packages for buster |
Patch for thumbor is in the works, needs to finish adding test cases which I'll try to do this weekend so it's ready to go out when ready.
Change 617874 had a related patch set uploaded (by AntiCompositeNumber; owner: AntiCompositeNumber):
[operations/software/thumbor-plugins@master] Add tests for MPEG-1 and MPEG-2 thumbnailing
Change 617874 merged by Gilles:
[operations/software/thumbor-plugins@master] Add tests for MPEG-1 and MPEG-2 thumbnailing
The JPG previews of the MPEGs don't seem to work either, for example https://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Relaisraum_im_Stellwerk_K%C3%B6ln_Kf.mpg/1024px--Relaisraum_im_Stellwerk_K%C3%B6ln_Kf.mpg.jpg.
As far as I know this worked fine in testing and if there's a problem in production, nobody has actually tested it or described what's going wrong in terms of error messages or logs. I don't know if it actually got deployed or not.
I also don't know who's responsible for running thumbor in production right now, or how to contact them.
Another five months passed, there are hundreds of MPEG videos on Commons, which have neither thumbnails nor an understandable explanation for why there is no thumbnail (just a generic 500 Internal Server Error message). When will it be fixed? Honestly I don’t know how MPEG uploads could have been enabled without proper support in the first place.
Removing task assignee due to inactivity, as this open task has been assigned for more than two years. See the email sent to the task assignee on February 06th 2022 (and T295729).
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome.
If this task has been resolved in the meantime, or should not be worked on ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator.
There's now active work on thumbor again, which should help us actually resolve this. :) I'm committed to help with some documentation on how to add type formats, and trying to bang the MPEG stuff back into shape.
I won't assign until I'm actively working on it though, probably early July after I finish up some other stuff.
Adding a note to this ticket: thumbor should apply deinterlacing on files that indicate interlacing in their video metadata; this will require a quick call into ffprobe.
I don't _think_ that originally got done, will be firing it up this afternoon to start looking it over so will double-check. :)
Can somebody change the subtask to "Bug"? I do not have enough permissions. Thank you!
OK, now that the new Thumbor is in production I picked on an MPEG file and manually re-triggered all its jobs, and they seem to have worked fine: https://commons.wikimedia.org/wiki/File:Anatomy_of_a_CC_License_(CC_Certificate_for_GLAM).mpg
So the remaining work here is to trigger this as a batch on the 490 files on https://commons.wikimedia.org/w/index.php?title=Special:MIMESearch/video/mpeg&limit=500&offset=0&mime=video%2Fmpeg and then we're done?
foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --video --mime=video/mpeg --missing --error --stalled --throttle I think should do it.
Nice! I don't know if you already issued that command or not, but some users are becoming really happy, reporting that thumbnails are starting to work - e.g.
https://commons.wikimedia.org/wiki/File:Esperimento_Torricelli_con_acqua.mpg
(Thanks @Galessandroni for waiting) Really thanks for this
Mentioned in SAL (#wikimedia-operations) [2023-05-17T16:58:50Z] <Guest4300> Running foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --video --mime=video/mpeg --missing --error --stalled --throttle on mwmaint1002 for T244570
@Jdforrester-WMF Wow, fantastic job. I can confirm that all the broken thumbnails I had seen are now generating!
OK, I'm declaring this Resolved as much as it's expected to be by the Thumbor upgrade; huge thanks to the Platform team for their work on this.
There appear to be a handful of "MPEG" videos that are not recognised still (e.g. https://commons.wikimedia.org/wiki/File:1998_BPA_conductor_gallop_in_eastern_Idaho.mpg), but that should be handled in different tasks.
Thanks seconded, it’s good to see MPEG is finally really supported!
There appear to be a handful of "MPEG" videos that are not recognised still (e.g. https://commons.wikimedia.org/wiki/File:1998_BPA_conductor_gallop_in_eastern_Idaho.mpg), but that should be handled in different tasks.
The issue with that file is either a bug in getID3, or that the file is broken:
$getID3 = new getID3(); $getID3->analyze( '1998_BPA_conductor_gallop_in_eastern_Idaho.mpg' )['mpeg']['video']['pixel_aspect_ratio']; // 0.0 $getID3->analyze( '1998_BPA_conductor_gallop_in_eastern_Idaho.mpg' )['mpeg']['video']['pixel_aspect_ratio_text']; // "forbidden"
… and our code multiplies the width by pixel_aspect_ratio before generating the thumbnail, so it tries to generate a zero-pixel-wide thumbnail, which of course won’t work. In any case, it’s not the thumbnail generation that fails here, but the metadata extraction, the thumbnail generation is broken only because the stored metadata is broken.
(Sorry for commenting here, but I’m not sure if this deserves a new task, or we should just say “don’t upload broken files”.)
