Page MenuHomePhabricator

Fix missing thumbnail for MPEG (.mpg) videos
Closed, ResolvedPublicBUG REPORT

Description

Upload of MPEG-2 videos was enabled in February 2020 on WMF cluster. Since that moment, no thumbnails were created for those uploads.

Example videos (all are involved):

Specific example:

https://commons.wikimedia.org/wiki/File:Relaisraum_im_Stellwerk_K%C3%B6ln_Kf.mpg

Error missing thumbnail mpeg.png (719×1 px, 106 KB)

Event Timeline

brion added a subscriber: brion.

Patch for thumbor is in the works, needs to finish adding test cases which I'll try to do this weekend so it's ready to go out when ready.

Change 617874 had a related patch set uploaded (by AntiCompositeNumber; owner: AntiCompositeNumber):
[operations/software/thumbor-plugins@master] Add tests for MPEG-1 and MPEG-2 thumbnailing

https://gerrit.wikimedia.org/r/617874

Change 617874 merged by Gilles:
[operations/software/thumbor-plugins@master] Add tests for MPEG-1 and MPEG-2 thumbnailing

https://gerrit.wikimedia.org/r/617874

@Gilles was this simply never deployed, or is there something else going on ?

As far as I know this worked fine in testing and if there's a problem in production, nobody has actually tested it or described what's going wrong in terms of error messages or logs. I don't know if it actually got deployed or not.

I also don't know who's responsible for running thumbor in production right now, or how to contact them.

It hasn't been deployed, as far as I can see.

Another five months passed, there are hundreds of MPEG videos on Commons, which have neither thumbnails nor an understandable explanation for why there is no thumbnail (just a generic 500 Internal Server Error message). When will it be fixed? Honestly I don’t know how MPEG uploads could have been enabled without proper support in the first place.

Removing task assignee due to inactivity, as this open task has been assigned for more than two years. See the email sent to the task assignee on February 06th 2022 (and T295729).

Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome.

If this task has been resolved in the meantime, or should not be worked on ("declined"), please update its task status via "Add Action… 🡒 Change Status".

Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator.

valerio.bozzolan renamed this task from Create thumbnail for MPEG videos to Create thumbnail for MPEG (.mpg) videos.Jun 22 2022, 2:55 PM
valerio.bozzolan updated the task description. (Show Details)

There's now active work on thumbor again, which should help us actually resolve this. :) I'm committed to help with some documentation on how to add type formats, and trying to bang the MPEG stuff back into shape.

I won't assign until I'm actively working on it though, probably early July after I finish up some other stuff.

Adding a note to this ticket: thumbor should apply deinterlacing on files that indicate interlacing in their video metadata; this will require a quick call into ffprobe.

I don't _think_ that originally got done, will be firing it up this afternoon to start looking it over so will double-check. :)

valerio.bozzolan renamed this task from Create thumbnail for MPEG (.mpg) videos to Fix missing thumbnail for MPEG (.mpg) videos.Oct 13 2022, 9:39 AM

Can somebody change the subtask to "Bug"? I do not have enough permissions. Thank you!

TheDJ changed the subtype of this task from "Task" to "Bug Report".Oct 13 2022, 10:16 AM
BCornwall changed the task status from Open to Stalled.Mar 13 2023, 6:34 PM
BCornwall added a subscriber: BCornwall.

OK, now that the new Thumbor is in production I picked on an MPEG file and manually re-triggered all its jobs, and they seem to have worked fine: https://commons.wikimedia.org/wiki/File:Anatomy_of_a_CC_License_(CC_Certificate_for_GLAM).mpg

So the remaining work here is to trigger this as a batch on the 490 files on https://commons.wikimedia.org/w/index.php?title=Special:MIMESearch/video/mpeg&limit=500&offset=0&mime=video%2Fmpeg and then we're done?

foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --video --mime=video/mpeg --missing --error --stalled --throttle I think should do it.

Nice! I don't know if you already issued that command or not, but some users are becoming really happy, reporting that thumbnails are starting to work - e.g.

https://commons.wikimedia.org/wiki/File:Esperimento_Torricelli_con_acqua.mpg

(Thanks @Galessandroni for waiting) Really thanks for this

Mentioned in SAL (#wikimedia-operations) [2023-05-17T16:58:50Z] <Guest4300> Running foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --video --mime=video/mpeg --missing --error --stalled --throttle on mwmaint1002 for T244570

Nice! I don't know if you already issued that command or not, but some users are becoming really happy, reporting that thumbnails are starting to work - e.g.

https://commons.wikimedia.org/wiki/File:Esperimento_Torricelli_con_acqua.mpg

(Thanks @Galessandroni for waiting) Really thanks for this

Running now; was in a meeting, sorry.

Running now; was in a meeting, sorry.

Why sorry? Thank you for your work!

@Jdforrester-WMF Wow, fantastic job. I can confirm that all the broken thumbnails I had seen are now generating!

OK, I'm declaring this Resolved as much as it's expected to be by the Thumbor upgrade; huge thanks to the Platform team for their work on this.

There appear to be a handful of "MPEG" videos that are not recognised still (e.g. https://commons.wikimedia.org/wiki/File:1998_BPA_conductor_gallop_in_eastern_Idaho.mpg), but that should be handled in different tasks.

OK, I'm declaring this Resolved as much as it's expected to be by the Thumbor upgrade; huge thanks to the Platform team for their work on this.

Thanks seconded, it’s good to see MPEG is finally really supported!

There appear to be a handful of "MPEG" videos that are not recognised still (e.g. https://commons.wikimedia.org/wiki/File:1998_BPA_conductor_gallop_in_eastern_Idaho.mpg), but that should be handled in different tasks.

The issue with that file is either a bug in getID3, or that the file is broken:

$getID3 = new getID3();
$getID3->analyze( '1998_BPA_conductor_gallop_in_eastern_Idaho.mpg' )['mpeg']['video']['pixel_aspect_ratio']; // 0.0
$getID3->analyze( '1998_BPA_conductor_gallop_in_eastern_Idaho.mpg' )['mpeg']['video']['pixel_aspect_ratio_text']; // "forbidden"

… and our code multiplies the width by pixel_aspect_ratio before generating the thumbnail, so it tries to generate a zero-pixel-wide thumbnail, which of course won’t work. In any case, it’s not the thumbnail generation that fails here, but the metadata extraction, the thumbnail generation is broken only because the stored metadata is broken.

(Sorry for commenting here, but I’m not sure if this deserves a new task, or we should just say “don’t upload broken files”.)