Fri, Jan 17
I suspect it's the file size limit yes -- ffmpeg seems to be exiting with SIGXFSZ (signal 25, exit code 153) which means 'exceeded file size'. This is probably not the file system limit, but the resource limit applied to the transcode process? However transcodes going over 4 GB may have trouble being stored anyway. :(
Seems to have been resolved in the meantime, probably by using updated ffmpeg on servers.
Guess this didn't seem that important, closing out.
All this ran, and later a bunch re-ran, some time ago. Closing out.
We chose to reimplement with oojs ui and have implemented.
New click-to-load mode behaves more like the old mode, and seems ok here. Closing out.
This seems to be largely dealt with the new load-on-click mode, I'm going to close it out.
Thu, Jan 16
Probably a side-effect of how the subtitle pages are specially rendered...
This seems to be due to a corruption problem in Photoshop or some other editing tool that produces a huge amount of extra junk data in the XMP metadata. You can find reports of this problem on forums like here: https://feedback.photoshop.com/photoshop_family/topics/corrupt-ancestors-tag-in-xmp-causing-giant-file-sizes-in-photoshop
Wed, Jan 15
The fields are currently defined in tables.sql as int unsigned which is a 32-bit type with a range from 0 to 2^32 - 1 (one byte shy of 4 GiB). To record sizes of files of 4 GiB or higher, redefining them as bigint unsigned, a 64-bit type, would be required.
Tue, Jan 14
Latest rev of https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/TimedMediaHandler/+/553442 includes a quick-fix for that: an inline transform property is added which adds the playsinline attribute and a mw-tmh-inline class which has the on-load transforms set up video.js for that player instead of turning it into a popup player trigger. This is then forced on for the embedded-iframe view.
(Well, not exactly broken, but ... kind of weird. ;) It forces the popup to run inside the iframe, which adds a title bar even at small sizes.)
Acknowledged, ?embedplayer=yes mode is indeed broken with those changes. ;_; I'll fix that shortly.
Mon, Jan 13
Native CPU-run VP9 seems pretty ok through 720p on my lower-spec machines, but YMMV. :) AV1 is much more CPU-intensive and I definitely don't recommend switching to it wholesale yet, especially at high resolutions.
Mon, Jan 6
Definitely saw it on https, on phab. The ipv6 timeout behavior makes it hard to see reliably so not sure if it was hitting other sites consistently but the packet loss on ping to load balancer was suspicious.
Dec 20 2019
Dec 19 2019
Thanks all! I'll try the email queue next time something crops up for future reference. :)
Dec 17 2019
Could remove it yeah Special:MediaStatistics does that top section better. Maybe stick in a link for good measure as a related info? I think Special:TimedMediaHandler should concentrate on the transcode status info (and maybe get renamed to match).
Dec 11 2019
sudo mtr -z -s 1000 -T -P 443 phabricator.wikimedia.org gives similar results:
Dec 10 2019
Looking... it looks like on success transcode_error is cleared but transcode_time_error is not. However it shouldn't be getting set in the first place unless something's awry...
Dec 5 2019
Ok, I'll run:
Ideally: fix the installs ASAP :) If can't be done: disable $wgFFmpegVP9RowMT and that should hopefully work if it reverted to a version that groks vp9 but doesn't know the -row-mt option.
Nov 27 2019
Updated https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/TimedMediaHandler/+/529819/ to use the better minified package. Much smaller JS payload now. :)
Ah, OSs shipping old stuff is always fun. :D No real danger to it, and it's clear enough in the code if we refactor and clean it out later. :)
@TheDJ let's try using it without wgMinimumVideoPlayerSize and drop it back in if we decide the always-transformation is confusing.
Nov 26 2019
I'm a bit reluctant to add explicit support for obsolete versions of ffmpeg, since it'll never get tested and other things that break it are likely to be added. Is there a reason you're running an out of date ffmpeg?
@AlexisJazz sorry about any frustration from the on-Commons discussion in January. It took some discussion of our own on Phabricator to decide what to do, both about the incorrect HTML security trigger and about the more general removal of obsolete checks, and communication wasn't and still isn't very consistent. Sometimes you get advice on-wiki that's based on the current general state of things, but the state of things changes -- we have indeed decided that the IE 6/7 content-type-sniffing vulnerability is no longer a serious security consideration, but in January we hadn't decided that yet so you were given advice based on the current state of the world.
Nov 15 2019
This is a compat problem with the old player; the new player mode is getting an update and should get merged soonish, and it's fully functional on mobile.
Nov 7 2019
Note that a Promise polyfill can be loaded at the cost of a few kb of compat code, but there's so many other reasons to want to deprecate IE 11 and to move to newer JS versions and most of the other things can't be polyfilled so easily.
Oct 21 2019
I have some more diagnostic info on T235934 for what appears to be the same problem (installing mobilefrontend). Workaround is to manually clone extensions/MobileFrontend and re-run vagrant provision.
Oct 18 2019
Yeah, looks like it's dying in the same place over on that one.
Oct 17 2019
Oct 15 2019
Oct 10 2019
Sounds like some of that sequence resulted in a conflict that didn't get handled well at the time. Following the error message back to the code symptom, the case is this in LocalFileRestoreBatch.php:
Oct 2 2019
Sep 26 2019
I've updated the documentation at https://www.mediawiki.org/wiki/MediaWiki-Vagrant to link to this bug report and mention that you can manually log in with vagrant ssh and run mwscript maintenance/changePassword.php --user=Admin --password=vagrant to set the initial password.
Sep 24 2019
Haven't encountered this in a while; Comcast etc may have improved the intermediate routes. Closing out as resolved.
Sep 19 2019
Looks like the audio wasn't getting initialized at the right time (how did it work on iOS? no clue!) unless mwEmbedSupport has already been loaded.
Sep 17 2019
Sep 11 2019
Looks like in PHP 7.2 the empty string gets interned to a special shared instance (zend_empty_string)... It's possible something's corrupting zend_empty_string, either its contents or its hash (h) field. If we can get a core dump it should be possible to inspect the symbol...
Aug 28 2019
Aug 21 2019
I'm still not sure I can reliably tell the difference but I trust you. :) Titles being sometimes late by random amounts up to 250ms seems like it would make sense given the poor latency guarantees on the timeupdate event, so I'm not sure there's much we can do directly here...
Aug 20 2019
Could this be latency from the timeupdate event? It can trigger with a wait as long as 250ms between events per spec.
Aug 14 2019
Note the parser cache key is not available to the hook. Fun!
I think default mode is checked because there's a separate pcache suffix for folks using the beta-mode -- this is targeted at clearing the non-beta-mode entries after a site-config switch.
The only POSTs I see in there are in setContainerAccess, addMissingHashMetadata, and doDescribeInternal, which all update existing files. Could be there's a hole in the logic of one of them that's dropping Content-Type, or that some files were missing it to begin with but it was being filled in on high-level fetches, or something, so they worked until something updated the data?
Hmm, I notice this in SwiftFileBackend.php:
Great, I'll tidy up this patch in a bit (once I get the subtitle lazy-loading patched back in) and we'll worry about lazy-loading later.
Since AFAIK extensions are meant to use wikipage.content for setup, I think any area where wikitext features are expected to work _should_ trigger the hook... A common class name would still require knowing when to run setup, and potentially one might want to update the indicator (what happens during edit preview?). So we either have to add a second hook that does the same thing as wikipage.content but is explicitly scoped to non-primary areas, or ...? Hmm, bears further thought.
For a video that triggers player setup but hasn't actually been played, in Chrome/Firefox where no ogv.js shim is needed, they seem to weigh in (as reported from mw.inspect, so uncompressed) at around 293 KiB for MwEmbed+jquery.ui and 288 KiB for video.js with the core.min.js variant.
Or... perhaps MW should run wikipage.content against the contents of <indicator> as well?
Looking at the TimedMediaHandler JS, it uses the wikipage.content hook which passes a jQuery object holding the content area in which to process elements. Anything outside the content area is not processed.
Can you create a copy of the template that uses the form that fails? I'm uncertain what to modify from the description.
Aug 12 2019
Would you want to trim in the base handling of the metadata, or in the output API serialization specifically? Might be best to do it all around if we're going to apply such rounding in ways that are machine-readable.
Patch switches to the core distribution (no http-streaming component) and the more aggressively pre-minified version. Not quite complete, as this means we can't patch in the text track lazy-loading on top as easily.
Aug 9 2019
This seems to be during a chunked upload of an Ogg audio or video file. Any chance we can get the original file chunks?
I can repro the inconsistency on both domains. I think it's a HHVM vs Zend PHP difference; the shorter serialization is served from HHVM 3.18.6 while the longer one is served from PHP 7.2.16.
Jul 29 2019
We can probably reduce the size for now by using the video.js core distribution (without the http-streaming component which we don't use yet) and using more aggressive minification (rather than using our own minifier, which doesn't shorted variable names and such).
Jul 26 2019
Jul 13 2019
Confirmed this fixes the transcodes; resolving.
Fixing the .ogv file by setting the 'ROTATE=180' to 'ROTATE=0':
I suspect that the 'rotate' annotation in the comments is not actually handled by Firefox, VLC, or ogv.js -- certainly I never wrote code in ogv.js to handle it -- and may be a leftover from the MP4/3gpp original file the .ogv was sourced from.
This may be a problem that's been since fixed in upstream ffmpeg...?
For me, the original plays upside-down in Chrome, right-side-up in Firefox, VLC, and ogv.js. So this ain't handled quite universally I suppose. ;) I'll take a peek.
Jul 12 2019
Jul 1 2019
Jun 29 2019
Jun 28 2019
Jun 27 2019
Note a deployment typo caused this fix to not get deployed in fact. Whoops! Roan's fixing it now.
Fix deployed. Should no longer cause fatal errors, but should log the transcoder output into the transcode table so we can review it later for suspicious commonalities.
Ok Reedy started this a while ago under T226713, so no need to re-run it a second time. It's probably the cause of seeing multiple fatals over on T226748, since all failure cases would come in a relatively short time. :D
Ah great, that'll do the job.
I have this ready to run on mwmaint1002 but want to wait until the fix for the fatal error in T226748 is deployed so I don't spam the error logs.
cleanupTranscodes.php should be able to do this I believe, I'll test offline later.
Jun 26 2019
(main difference from previous behavior is that in practice most videos got the image with popup link where now they get a video directly)
Ehhh, it's not hard to add a fallback image link that points to the file page. But I wouldn't necessarily block on it if it's not done soon.
Jun 24 2019
Note that with the "New video player" beta feature enabled, this can be tested in production -- just make sure you're logged in in the mobile browser and it will pick up the preference.
Probably should adjust the preload setting; ISTR some related issue in past. Will test further once no longer encountering unrelated server errors.
Jun 23 2019
Yeah, looks like we only read 1024 header bytes and it appears later in both those files, around 4000+ bytes in.
Confirmed that it's resolved with the video,js player frontend ("New video player" in beta features preferences)
Jun 22 2019
I was able to reproduce this locally using the files via InstantCommons, and it looks like the root problem is that some of the files are misidentifed as audio, causing the packed-gallery code to use a small icon width instead of a large width constraint here: