Page MenuHomePhabricator
Feed Advanced Search

Apr 22 2020

Don-vip awarded T103062: Please add www.dvidshub.net to the wgCopyUploadsDomains whitelist of Wikimedia Commons a Love token.
Apr 22 2020, 11:12 AM · Patch-For-Review, Commons, Wikimedia-Site-requests

Jun 14 2017

Revent added a comment to T167815: Conduct MP3 patrol discussion.

In my experience (and I have somewhat failed in this at the past) the best method would be to start a simple 'discussion' about the perceived issue, suggest a solution, and then ask the community to debate the matter. Once there is at least some degree of consensus about the correct solution, then ask the community to actually vote on approving it.

Jun 14 2017, 9:59 PM · Readers-Community-Engagement, Community-Relations-Support (Jul-Sep 2017), Commons, Multimedia
Revent added a comment to T120288: Enable MP3 uploads on Wikimedia Commons and TMH playback.

@CKoerner_WMF I think the most important thing that needs to be understood here is that any decision to enable mp3 uploading on Commons that is not implemented as the result of a explicit decision by the Commons community is likely to end badly. Now that the legal issues have been cleared up, and it's established that the technical implementation is not a problem, most of this discussion really needs to move onwiki, so that the community as a whole will have an 'investment' in the process of deciding how it is implemented. If a plan such as yours (which is not bad) is presented to the community as a 'fait accompli', even if merely for discussion, there will be grumbling.

Jun 14 2017, 8:41 PM · User-notice-archive, Patch-For-Review, Trust-and-Safety, TimedMediaHandler, WMF-Legal, UploadWizard, Commons, Multimedia

May 30 2017

TheDJ awarded T166482: Persistent failure of TMH to transcode videos at specific resolutions a Doubloon token.
May 30 2017, 11:11 AM · SRE-swift-storage, SRE, TimedMediaHandler
Revent added a comment to T166482: Persistent failure of TMH to transcode videos at specific resolutions.

@fgiunchedi Thanks for jumping onto this so quickly. I'll start working on resetting the others, and let you know if the problem still exists.

May 30 2017, 11:03 AM · SRE-swift-storage, SRE, TimedMediaHandler

May 29 2017

Revent added a comment to T166482: Persistent failure of TMH to transcode videos at specific resolutions.

@fgiunchedi Much smaller files have the same problem. See https://commons.wikimedia.org/wiki/File:Janusz_Cedro_live_in_concert-_Amazing_Grace.ogv for example, which is only 17MB.

May 29 2017, 9:51 PM · SRE-swift-storage, SRE, TimedMediaHandler
Revent added a comment to T163068: More missing 'original' files on Commons.

Some new examples (with audio files)...

May 29 2017, 12:59 AM · SRE, SRE-swift-storage, Commons
Revent added a comment to T120288: Enable MP3 uploads on Wikimedia Commons and TMH playback.

@Koavf TinEye reverse search (which is a gadget on Commons) handles cropped images quite well. It's not available to us as an 'automatic' method, due to cost, but they allow a number of free searches per day....using the gadget, they are counted against the specific requesting user, not the WMF or labs.

May 29 2017, 12:43 AM · User-notice-archive, Patch-For-Review, Trust-and-Safety, TimedMediaHandler, WMF-Legal, UploadWizard, Commons, Multimedia
Revent added a comment to T132650: Copyright detection (acoustic fingerprint matching) for audio files.

@Fastily I think the main concern here, really, is simply Commons.... for any other wiki, the volume of 'valid' audio or video uploads should be sufficiently low that abuse would be easily recognizable.

May 29 2017, 12:34 AM · Technical-Tool-Request, Commons
Revent created T166482: Persistent failure of TMH to transcode videos at specific resolutions.
May 29 2017, 12:02 AM · SRE-swift-storage, SRE, TimedMediaHandler
Revent added a comment to T163068: More missing 'original' files on Commons.

I fixed two of those by uploading a new copy from youtube (the source), and got the author (Anna Frodesiak) to upload a new copy of another one.

May 29 2017, 12:00 AM · SRE, SRE-swift-storage, Commons

May 17 2017

Revent added a comment to T154100: Implement Internet Archive BookReader in Commons & Wikisource.

While Commons isn't specifically intended for reading books, a nice way to do so would be a definite plus (and the IA's tool is awesome). Perhaps it could be setup to load instead of MediaViewer when looking at multipage files.

May 17 2017, 9:14 AM · Wikimedia-Hackathon-2017, Community-Wishlist-Survey-2016, Internet-Archive, All-and-every-Wikisource, Commons
Revent awarded T154100: Implement Internet Archive BookReader in Commons & Wikisource a Love token.
May 17 2017, 9:12 AM · Wikimedia-Hackathon-2017, Community-Wishlist-Survey-2016, Internet-Archive, All-and-every-Wikisource, Commons

May 8 2017

Revent updated subscribers of T163535: Upload verification-error possibly triggered by EXIF.

@matmarex That's truly esoteric.

May 8 2017, 9:10 AM · MediaWiki-Uploading, Multimedia, Commons

Apr 25 2017

Revent added a comment to T161836: 404 error while accessing some images files (e.g. djvu, jpg, png, webm) on Commons and other sites.

https://commons.wikimedia.org/wiki/File:Autonomous_bus_trials_South_Perth_-_3.ogv
https://commons.wikimedia.org/wiki/File:Cheek_piercing_at_a_ritual_in_Qionghai,_Hainan,_China.ogv

Apr 25 2017, 11:27 PM · User-fgiunchedi, SRE-swift-storage, SRE, Multimedia

Apr 21 2017

Revent added a comment to T163535: Upload verification-error possibly triggered by EXIF.

@Fae https://commons.wikimedia.org/wiki/Commons:FAQ#What_does_the_upload_error_This_file_contains_HTML_or_script_code_that_may_be_erroneously_interpreted_by_a_web_browser._mean.3F is relevant... the error, apparently, is actually that it's not detecting them all.

Apr 21 2017, 11:55 AM · MediaWiki-Uploading, Multimedia, Commons

Apr 20 2017

Revent added a comment to T113191: Moving a video file resets the transcodes to "Unknown status".

I'll give some examples when I next see it.

Apr 20 2017, 7:31 PM · Patch-For-Review, TimedMediaHandler-Transcode
Revent added a comment to T113191: Moving a video file resets the transcodes to "Unknown status".

@brion Dug this up out of the old stuff. Based on recent logs, this still happens... since we repurged all the file pages of videos with no transcodes, I have been watching the move log and resetting the recent moves.

Apr 20 2017, 7:31 PM · Patch-For-Review, TimedMediaHandler-Transcode

Apr 17 2017

Revent added a comment to T157028: Many transcodes still fail.

This is now down to less than 400 transcodes.

Apr 17 2017, 2:46 PM · WMF-General-or-Unknown, TimedMediaHandler-Transcode

Apr 16 2017

Revent created T163068: More missing 'original' files on Commons.
Apr 16 2017, 10:34 AM · SRE, SRE-swift-storage, Commons
Revent added a comment to T132650: Copyright detection (acoustic fingerprint matching) for audio files.

To clarify my last comment, if we enable mp3 we are going to be swamped with massive amounts of copyvios.

Apr 16 2017, 10:28 AM · Technical-Tool-Request, Commons

Apr 11 2017

Revent added a comment to T162395: Add .mp3 to the list of accepted file types on Wikimedia Commons uploads.

The enwiki article claims the last US patent on mp3 expires on 16 April 2017, though it's not referenced other than to the patents themselves.

Apr 11 2017, 4:49 PM · WMF-Legal, Commons
Revent added a comment to T132650: Copyright detection (acoustic fingerprint matching) for audio files.

Oh gawd.

Apr 11 2017, 4:41 PM · Technical-Tool-Request, Commons
Revent closed T162689: Files in one category when they are categorized in another as Resolved.

All done, and in the correct category now.

Apr 11 2017, 2:08 PM · Commons
Revent claimed T162689: Files in one category when they are categorized in another.
Apr 11 2017, 2:07 PM · Commons
Revent added a comment to T162689: Files in one category when they are categorized in another.

@Ivanhercaz I wrote a quick script to use the API to 'forcelinkupdate' the 400-odd files. It's nearly done now.

Apr 11 2017, 2:02 PM · Commons
Revent added a comment to T162689: Files in one category when they are categorized in another.

@Ivanhercaz Purging the categories does not make any difference, because category membership is a property of the file pages themselves, not the categories. The solution is to either wait for the job queue to catch up, or to force it by performing a null edit to each file page. See https://www.mediawiki.org/wiki/Manual:Purge#Null_edits.

Apr 11 2017, 1:33 PM · Commons

Apr 5 2017

Revent added a comment to T161836: 404 error while accessing some images files (e.g. djvu, jpg, png, webm) on Commons and other sites.

https://commons.wikimedia.org/wiki/File:Walking_Keage_Incline.webm reappeared, and has been reset

Apr 5 2017, 3:18 AM · User-fgiunchedi, SRE-swift-storage, SRE, Multimedia

Apr 1 2017

Revent added a comment to T161836: 404 error while accessing some images files (e.g. djvu, jpg, png, webm) on Commons and other sites.

Of the ones I listed before (videos), these are now back...
https://commons.wikimedia.org/wiki/File:X5Flare_AIA193.webm
https://commons.wikimedia.org/wiki/File:Wagejot-4804259.webm
https://commons.wikimedia.org/wiki/File:T%C5%AFn%C4%9B_pod_mal%C3%BDm_vodop%C3%A1dem_Peri%C4%8Dn%C3%ADk_a_koup%C3%A1n%C3%AD.webm
https://commons.wikimedia.org/wiki/File:%D0%A1%D0%92-%D0%94%D0%9D%D0%A0-564._%D0%9D%D0%BE%D0%B2%D1%8B%D0%B9_%D0%B3%D0%BE%D0%B4_%D0%B2_%D0%94%D0%BE%D0%BD%D0%B5%D1%86%D0%BA%D0%B5.webm

Apr 1 2017, 11:54 PM · User-fgiunchedi, SRE-swift-storage, SRE, Multimedia
Revent added a comment to T161836: 404 error while accessing some images files (e.g. djvu, jpg, png, webm) on Commons and other sites.

The current list of affected files (at least, of ones with a failed transcode that make them apparent) is....
https://commons.wikimedia.org/wiki/File:X5Flare_AIA193.webm
https://commons.wikimedia.org/wiki/File:Wagejot-4804259.webm
https://commons.wikimedia.org/wiki/File:Walking_Keage_Incline.webm
https://commons.wikimedia.org/wiki/File:T%C5%AFn%C4%9B_pod_mal%C3%BDm_vodop%C3%A1dem_Peri%C4%8Dn%C3%ADk_a_koup%C3%A1n%C3%AD.webm
https://commons.wikimedia.org/wiki/File:%D0%A1%D0%92-%D0%94%D0%9D%D0%A0-564._%D0%9D%D0%BE%D0%B2%D1%8B%D0%B9_%D0%B3%D0%BE%D0%B4_%D0%B2_%D0%94%D0%BE%D0%BD%D0%B5%D1%86%D0%BA%D0%B5.webm

Apr 1 2017, 1:20 PM · User-fgiunchedi, SRE-swift-storage, SRE, Multimedia
Revent updated the task description for T160800: TimedMediaHandler doesn't prevent blocked users from restarting transcodes (CVE-2022-28211).
Apr 1 2017, 1:08 PM · MW-1.39-notes (1.39.0-wmf.6; 2022-04-04), SecTeam-Processed, Security-Team, Vuln-Infoleak, Security, TimedMediaHandler-Transcode, Vuln-DoS
Revent added a comment to T141704: Unable to delete certain files due to "inconsistent state within the internal storage backends".

@Ankry Just now, for me at least, that link works. The errors seem to be intermittent.

Apr 1 2017, 1:05 PM · MediaWiki-File-management, Wikimedia-production-error, User-fgiunchedi, Multimedia, Commons, SRE, SRE-swift-storage
Revent added a comment to T161918: videoscalers (mw1168, mw1169) - high load / overheating.

I reset the transcodes that had been running for egregiously long periods... this included ones running for 10+ hours that should have run in a minute or two. The transcodes then showed up in https://quarry.wmflabs.org/query/17726. The list was initially 90 or so... most have since 'reappeared', and I have then reset them to get the transcodes done.

Apr 1 2017, 11:51 AM · SRE
Revent added a comment to T161836: 404 error while accessing some images files (e.g. djvu, jpg, png, webm) on Commons and other sites.

The video 'works' because when you simply play it, you view some transcode based on your preferences. If you try to view the original video by clicking the link directly under the thumbnail, you get a 404.

Apr 1 2017, 11:47 AM · User-fgiunchedi, SRE-swift-storage, SRE, Multimedia

Mar 28 2017

Revent awarded T161517: Allow anonymous users to change interface language on Commons with ULS a Manufacturing Defect? token.
Mar 28 2017, 5:05 PM · [DEPRECATED] wdwb-tech, Traffic-Icebox, SRE, Commons, Wikimedia-Site-requests, I18n

Mar 19 2017

Revent added a comment to T151199: Please mass-reset the video transcodes of tens of thousand videos stuck in "Unknown" state.

I have actually been doing this (through the API) in reasonable-sized batches for several days now.

Mar 19 2017, 3:30 AM · Wikimedia-Site-requests, Wikimedia-maintenance-script-run, TimedMediaHandler-Transcode

Mar 17 2017

Revent added a comment to T160800: TimedMediaHandler doesn't prevent blocked users from restarting transcodes (CVE-2022-28211).

I had emailed Brion about this, after a mention on IRC.

Mar 17 2017, 11:10 PM · MW-1.39-notes (1.39.0-wmf.6; 2022-04-04), SecTeam-Processed, Security-Team, Vuln-Infoleak, Security, TimedMediaHandler-Transcode, Vuln-DoS

Feb 18 2017

Revent created T158464: Wm-bot recent changes is not visible from IRC.
Feb 18 2017, 5:42 AM · WM-Bot

Feb 17 2017

Revent added a comment to T158352: Flickr Public Domain Mark.

But I would think that the main evidence for such a consensus would be that such files have been marked for 'delayed speedy' for close to two years now.

Feb 17 2017, 4:53 PM · Multimedia, UploadWizard
Revent added a comment to T158352: Flickr Public Domain Mark.

@Aklapper There was an RFC in 2015... https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Flickr_and_PD_images

Feb 17 2017, 4:15 PM · Multimedia, UploadWizard
Revent added a comment to T158352: Flickr Public Domain Mark.

@matmarex If the UW is actually requiring the uploader to input a license tag other than the PDM, then it's probably fine.... this came up at a DR, and possibly I misunderstood exactly what the tool was doing. There is a consensus on Commons that the PDM 'in and of itself' is not an acceptable license, but it sounds like you are saying that the tool requires the uploader to do what the template itself would ask for. If that's the case, then we can close this as invalid I guess.

Feb 17 2017, 2:17 AM · Multimedia, UploadWizard

Feb 16 2017

Revent created T158352: Flickr Public Domain Mark.
Feb 16 2017, 8:39 PM · Multimedia, UploadWizard

Feb 15 2017

Revent added a comment to T158137: Renaming audio/video files leaves orphaned transcode table entries that clog reports.

See https://quarry.wmflabs.org/query/14842

Feb 15 2017, 6:49 PM · Video, TimedMediaHandler-Transcode

Feb 10 2017

Revent added a comment to T157763: Replace Apple Maps with a freely licensed alternative .

Leaflet (using OSM data) would be a better solution.

Feb 10 2017, 2:43 AM · User-Josve05a, Maps, WMF-Legal, iOS-app-feature-Places, Wikipedia-iOS-App-Backlog

Feb 9 2017

Revent added a comment to T108234: Run a bot to add 240p webm/ogv, re-run 360p/480p ogv video transcodes.

@brion There are also many (a little over 15k) videos with missing 160.webm transcodes. See https://quarry.wmflabs.org/query/15792.

Feb 9 2017, 10:32 PM · Patch-For-Review, Video
Revent added a comment to T157194: Redirects listed as queued transcodes.

@TheDJ See the hits of https://quarry.wmflabs.org/query/14916. The first 40 or so are similar issues (transcodes of redirects that won't go away). That list had originally had, IIRC, about a thousand, but the vast majority were, like I said, 'easily fixable'. I haven't looked at the ones that showed up on it since Dispenser reset everything.

Feb 9 2017, 10:26 PM · TimedMediaHandler-Transcode

Feb 8 2017

Revent added a comment to T157621: Special:TimedMediaHandler does not exist and won't even load a webpage.

There are 139k in the queue, and 112k unitialized... that's surely the cause.

Feb 8 2017, 11:05 PM · MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), Wikimedia-production-error, TimedMediaHandler, Commons
Revent added a comment to T157194: Redirects listed as queued transcodes.

@TheDJ Steinsplitter had moved a 'large' number of videos from .ogg to .ogv, and for these two there were entries left in the transcode table for the redirects. Back when I had started working on cleaning up the many transcode issues months ago, there were many such files..., it seemed to be because the file was moved 'while' it had running transcodes, and those transcodes then errored out because the file no longer existed, or because the file was deleted while it was still running.

Feb 8 2017, 8:58 PM · TimedMediaHandler-Transcode

Jan 26 2017

Revent added a watcher for TimedMediaHandler-Transcode: Revent.
Jan 26 2017, 11:26 AM
Revent added a comment to T156185: Big transcodes fail with Exitcode: 137 - SIGKILL.

The relevant table line is https://quarry.wmflabs.org/query/15801 <- error message was 'timeout'

Jan 26 2017, 11:26 AM · TimedMediaHandler-Transcode

Jan 25 2017

Revent added a comment to T154186: Certain videos on Commons do not start at all or have interruptions.

@TheDJ The search checks for the existence of 'not null' in transcode_time_success and transcode_time error. The ones that were originally in the report, other than the 35 there now, were all from 'years ago', and the ones I checked at the time were rather 'obviously' deleted shortly after being uploaded. As it stands now, if the file is deleted or renamed shortly after upload, the entries in the transcode table go away, so that seems to have been long ago fixed.

Jan 25 2017, 9:06 AM · Kaltura player, Video, Commons
Revent added a comment to T138967: Labs database replica drift.

I was asked to mention this here, although I'm unsure if it's actually a 'replication' issue.

Jan 25 2017, 1:54 AM · Data-Services, Epic, DBA
Revent added a comment to T154186: Certain videos on Commons do not start at all or have interruptions.

There is an issue that occurs sometimes (such as when the servers are restarted) where transcodes end up with both a 'success' and a 'failure' status in the SQL... they are shown on the file pages as successfully transcoded, but it seems that, in general, they were not.

Jan 25 2017, 1:11 AM · Kaltura player, Video, Commons

Jan 15 2017

Liuxinyu970226 awarded T153488: Commons video transcoders have over 6500 tasks in the backlog. a 100 token.
Jan 15 2017, 12:56 AM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Jan 14 2017

Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

@brion Yes, the 'pending' queue is now empty, other than brief spikes (mainly due to spurts of large uploads, or me throwing old failed stuff back through) and it re-empties itself in a reasonable time.

Jan 14 2017, 8:49 AM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Jan 13 2017

Revent awarded T155098: Rework job queue usage for TimedMediaHandler (video scalers) a Love token.
Jan 13 2017, 11:16 AM · MW-1.29-release (WMF-deploy-2017-02-07_(1.29.0-wmf.11)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), Patch-For-Review, TimedMediaHandler-Transcode

Jan 12 2017

Liuxinyu970226 awarded T153488: Commons video transcoders have over 6500 tasks in the backlog. a Heartbreak token.
Jan 12 2017, 3:33 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Jan 10 2017

Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

Just to update, the transcoders are still (as of now) mainly processing old multi-GB files.

Jan 10 2017, 10:55 AM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Jan 8 2017

Revent added a comment to T154733: Temporarily unassign 'transcode-reset' user right from Commons autoconfirmed and sysop groups.

@tomasz I think that it's important to prevent anything but 'new uploads' being thrown in the queue at least until it's actually completely caught up, and it's hard to estimate because it will depend on just how long the remaining ones take to be run. The queue is now down to 1393 transcodes, but it's also effectively been sorted by all this to many of the worst 'time intense' transcodes.... I have seen probably close to a dozen in there that are over 3GB in size. They will probably fail, but they need time to do so to get out the the queue.

Jan 8 2017, 8:59 PM · Wikimedia-Site-requests, Patch-For-Review, Video, TimedMediaHandler-Transcode, Commons
Revent added a comment to T154733: Temporarily unassign 'transcode-reset' user right from Commons autoconfirmed and sysop groups.

@Jasonanaggie There are other issues with the special page not correctly showing some tasks that are actually 'running'. Transcodes are being run, and the queue is going down noticeably, it's just that a significant number of the remaining ones are multi-GB files.

Jan 8 2017, 6:43 AM · Wikimedia-Site-requests, Patch-For-Review, Video, TimedMediaHandler-Transcode, Commons

Jan 6 2017

Revent added a comment to T154186: Certain videos on Commons do not start at all or have interruptions.

@hoo FYI, due to a recent operations issue, all of the backlogged video transcodes were booted out of the queue, and have to be restarted (at a sane rate, ofc)... I'm booting your videos back through first.

Jan 6 2017, 7:49 PM · Kaltura player, Video, Commons

Dec 27 2016

czar awarded T153488: Commons video transcoders have over 6500 tasks in the backlog. a The World Burns token.
Dec 27 2016, 8:04 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Dec 26 2016

Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

@elukey Just to be clear, I have not (other than possibly incidentally) been putting old failed transcodes back on the queue since the backlog exploded (my comment there was back in mid-November). I have been resetting the transcodes with 'broken' (as in, both queued and errored) DB statuses that were created when the servers were reset on the 19th, but those don't 'add anything' to the backlog.... they were already queued.

Dec 26 2016, 7:31 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

@Yann I have the impression that more action will be taken after the holidays.

Dec 26 2016, 4:12 AM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Dec 22 2016

Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

Jobs seem to be processing without problems... I have seen tasks completing successfully in significant numbers (hundreds), and very few recent failures.

Dec 22 2016, 11:25 AM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a comment to T153852: Creation of derivative video files is not working.

@Pokefan95 When I said 1920P, https://commons.wikimedia.org/wiki/File:Moscow_Ring_Railway_full_trip_-_view_from_ES2G_train.webm is actually the one I was thinking of... which yes, is not actually 1920P (which is not a real thing), but it is a 2.5GB full-HD video... I was vaguely making the point of 'insanely large files' in a very generic manner.

Dec 22 2016, 3:40 AM · SRE, Video, TimedMediaHandler, Commons

Dec 21 2016

Revent closed T150158: "Broken" video transcodes are sometimes not successfully detected as broken as Invalid.
Dec 21 2016, 12:58 PM · Commons, Video
Revent added a comment to T150158: "Broken" video transcodes are sometimes not successfully detected as broken.

This seems to have been subsumed into later tasks. I'll reopen a new task for the specific issues I'm still seeing.

Dec 21 2016, 12:58 PM · Commons, Video
Revent added a comment to T153852: Creation of derivative video files is not working.

This is known, and being worked. It's simply that the backlog became extremely large due to a high number of 'huge' (1920P, and an hour long) and a (resolved) bug that caused the servers to try to start an insane number of concurrent tasks. More transcoders have been added to the cluster, and the backlog should fall over time. Transcodes marked as "Added to Job queue [INVALID] ago" are indeed being run, there are just 10k tasks in the queue.

Dec 21 2016, 12:45 PM · SRE, Video, TimedMediaHandler, Commons

Dec 19 2016

Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

I appreciate the thanks...really. I'd already been working on kicking failed transcodes back through at a 'sane' rate, (on a mass scale, I mean, I had eliminated the 15k 'unitialized' transcodes, plus others that were added when the file pages were purged, presumable due to new targets being added) when this started. Since I'm not a 'coder-type', I can't really grasp the logic that is used to choose what to start next, I'm just basing my actions on the experience I have gained from actually 'using' the system. I do understand SQL, though, enough to watch the entries in the actual transcode table and how they are changed as things run.

Dec 19 2016, 8:03 AM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

To update, the backlog is now over 8000... when around, I have been kicking 'excessively large' (the rule of thumb I have been using is HD, and over 200MB) transcodes off the processing queue, by resetting them. This causes them to fail around half an hour later, instead of the 7+ hours it would take otherwise, and after doing so consistently for 5-6 hours the queue indeed starts to fall, but then I go to sleep and it goes back up by even more. The admin UI for TimedMediaHandler really should provide a more meaningful way to manage running transcodes.

Dec 19 2016, 7:42 AM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Dec 16 2016

Revent added a subtask for T152938: Server side upload for Jasonanaggie: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:54 PM · Server-side-upload-request, Commons
Revent added a subtask for T152943: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:54 PM · Commons, Wikimedia-Site-requests
Revent added parent tasks for T153488: Commons video transcoders have over 6500 tasks in the backlog.: T153357: Please upload large file to Wikimedia Commons, T153137: Please upload large file to Wikimedia Commons, T153136: Please upload large file to Wikimedia Commons, T152969: Please upload large file to Wikimedia Commons, T152943: Please upload large file to Wikimedia Commons, T152938: Server side upload for Jasonanaggie.
Dec 16 2016, 8:54 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a subtask for T153136: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:54 PM · Commons, Wikimedia-Site-requests
Revent added a subtask for T152969: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:54 PM · Commons, Wikimedia-Site-requests
Revent added a subtask for T153137: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:54 PM · Commons, Wikimedia-Site-requests
Revent added a subtask for T153357: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:54 PM · Commons, Wikimedia-Site-requests
Revent removed a parent task for T152938: Server side upload for Jasonanaggie: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:52 PM · Server-side-upload-request, Commons
Revent removed subtasks for T153488: Commons video transcoders have over 6500 tasks in the backlog.: T152938: Server side upload for Jasonanaggie, T152943: Please upload large file to Wikimedia Commons, T152969: Please upload large file to Wikimedia Commons, T153136: Please upload large file to Wikimedia Commons, T153137: Please upload large file to Wikimedia Commons, T153357: Please upload large file to Wikimedia Commons.
Dec 16 2016, 8:52 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent updated subscribers of T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:33 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

I'm 'owning' this because I'm taking the (annoying) responsibility of sitting on the Commons transcode queue, and trying to get everything that's 'pending' actually done.

Dec 16 2016, 8:33 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent claimed T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 8:31 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

So...

  • Video scalers are insufficiently powerful to transcode the 'very large' files people are uploading now.
  • TimedMediaHandler will continue starting transcodes until the server runs out of memory, instead of considering number of CPUs.
  • Resetting a running task does not immediately kill the running transcode
    • ^^ will both indicate a transcode is 'error' on the file page, and list it as 'queued' on the TMH special page.
    • ^^ will (apparently if the transcode completes before it dies) sometimes result in an entry in the transcode table with both a 'time of success' and a 'error time'. Such entries are shown with a 'negative number of seconds' as how long they took to transcode on the file page.
      • https://quarry.wmflabs.org/query/14728
      • The entries here are not examples of this, they are mainly old entries for files that were renamed. I 'fixed' the examples that were there by resetting them.
  • When a transcode is reset while running, and then 'fails', it creates an entry in the transcode table that is both 'an error' and 'queued'. This is actually useful, even though obviously not a design feature, as it has allowed removing large files from the queue without simply having them restart. When those files are 'again' reset (removing the error message) they go back to the 'head' of the processing queue, apparently... at least, they are the next started, ahead of the other 6500 pending transcodes.
  • Moving files does not always remove entries from the transcode table.
  • Deleting files does not always remove entries from the transcode table. Undeleting, and then redeleting, the files does.
Dec 16 2016, 8:27 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a comment to T153488: Commons video transcoders have over 6500 tasks in the backlog..

[1:31pm] Revent: Dereckson: PLEASE do not, at this time, run more server-side video uploads.
[1:31pm] Revent: PLEASE!
(snip)
[1:31pm] Revent: In brief?
[1:31pm] Revent: https://ganglia.wikimedia.org/latest/?r=month&cs=&ce=&c=Video+scalers+eqiad&h=&tab=m&vn=&hide-hf=false&m=cpu_report&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name
[1:32pm] Revent: There are over 6500 transcodes sitting in the queue.
[1:33pm] Revent: Most will run for 7 hours or so, and then time out… and the machine will try to run 100 at a time, and not process ANYTHING successfully.
(snip)
[1:33pm] Revent: There are a couple of tasks about needing to upgrade them.
[1:34pm] Revent: I’ve been messing with it, getting stuff to actually run, and in the process discovered several ‘more’ bugs in TimedMediaHandler.
(snip)
[1:35pm] Revent: The code apparently defines ‘capacity’ as ‘amount of ram’, not ‘number of CPUs.
(expression of dismay)
[1:35pm] Revent: You can’t multitask transcodes.
[1:36pm] Revent: (at least, not helpfully…. it’s a one-per-core thing)
[1:37pm] Revent: In brief…. I have been exploiting ‘another’ bug, by ‘resetting’ transcodes that are in the process of running, to halt the transcodes of all files over 200MB to let the servers at least catch up with the ‘normal’ business.
[1:38pm] Revent: It puts them back in the queue, but (based on how I read what it does to the DB) does not immediately halt the task, the task fails when it finds it’s ‘working files’ went away.
[1:39pm] Revent: It then adds an error message to the file in the transcode table, etc, which apparently prevents it from getting reprocessed even though it’s ‘queued'.
[1:39pm] Revent: It shows as ‘error’ on the file page, even tho it’s listed in the count of the queue.
[1:40pm] Revent: resetting it again makes it run, but puts it at the ‘front’ of the queue for some reason.
[1:40pm] Revent: (this has all been rather annoying, but has at least been getting ‘some’ files run, and brought the queue down some, slowly.
[1:41pm] Revent: I’m watching what I’m actually ‘doing’ with quarry, so I actually know which ones need reset (tho they show up in the queue anyhow)
[1:42pm] Revent: But, please, please, please, don’t dump 1000 more transcodes on the queue, lol. At least not right now.
[1:43pm] Revent: Dereckson: ^ hopefully all that makes some degree of sense.

Dec 16 2016, 8:10 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a parent task for T152938: Server side upload for Jasonanaggie: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 7:57 PM · Server-side-upload-request, Commons
Revent added a subtask for T153488: Commons video transcoders have over 6500 tasks in the backlog.: T152938: Server side upload for Jasonanaggie.
Dec 16 2016, 7:57 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a parent task for T152943: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 7:56 PM · Commons, Wikimedia-Site-requests
Revent added a subtask for T153488: Commons video transcoders have over 6500 tasks in the backlog.: T152943: Please upload large file to Wikimedia Commons.
Dec 16 2016, 7:56 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a parent task for T152969: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 7:55 PM · Commons, Wikimedia-Site-requests
Revent added a subtask for T153488: Commons video transcoders have over 6500 tasks in the backlog.: T152969: Please upload large file to Wikimedia Commons.
Dec 16 2016, 7:55 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a parent task for T153136: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 7:54 PM · Commons, Wikimedia-Site-requests
Revent added a parent task for T153137: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 7:53 PM · Commons, Wikimedia-Site-requests
Revent added a subtask for T153488: Commons video transcoders have over 6500 tasks in the backlog.: T153137: Please upload large file to Wikimedia Commons.
Dec 16 2016, 7:53 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent added a parent task for T153357: Please upload large file to Wikimedia Commons: T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 7:53 PM · Commons, Wikimedia-Site-requests
Revent added a subtask for T153488: Commons video transcoders have over 6500 tasks in the backlog.: T153357: Please upload large file to Wikimedia Commons.
Dec 16 2016, 7:53 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons
Revent created T153488: Commons video transcoders have over 6500 tasks in the backlog..
Dec 16 2016, 7:51 PM · User-notice-archive, User-Elukey, User-Joe, SRE, Video, TimedMediaHandler-Transcode, Commons

Nov 16 2016

Revent added a comment to T150605: Publish an analysis of the OurMine hack.

"Maury Markowitz" was compromised on enwiki within the last hour or so, the drama continues. See https://en.wikipedia.org/wiki/Special:Contributions/Maury_Markowitz

Nov 16 2016, 1:54 AM · Sustainability (Incident Followup), Security-Team

Nov 15 2016

Revent added a comment to T150067: Extend capacity for video scalers.

Just as an 'update', after being educated on how to use Quarry to look at older parts of the broken transcode list, I've been working on poking ones through the queue again. The ones it's showing me are from June of 2013... many are 'short' files, that complete successfully within less than a minute once kicked back in the queue. I've pushed several hundred (I did not keep exact track) back through already.

Nov 15 2016, 4:10 AM · hardware-requests, WMF-General-or-Unknown, SRE

Nov 13 2016

Revent added a comment to T150067: Extend capacity for video scalers.

Ok. As it now stands, all of the broken transcodes 'exposed' by TimedMediaHandler on Commons (I mean, the list at https://commons.wikimedia.org/wiki/Special:TimedMediaHandler ) are either...
A. Subjects of a bug that prevents 'any' successful transcode.

B. Subjects of a bug that prevents transcoding from OGV to WebM.

or C. Very long (over an hour) and large (from ~750k up to about 3G) that have simply failed to transcode after repeated attempts, even when run 'one at a time'. These often error out after 5-6 hours.

Nov 13 2016, 1:21 PM · hardware-requests, WMF-General-or-Unknown, SRE