Page MenuHomePhabricator

Run a bot to add 240p webm/ogv, re-run 360p/480p ogv video transcodes
Closed, ResolvedPublic

Description

Now that .ogv transcode settings have changed, all the existing 360p and 480p .ogv transcodes should be re-run, and the missing 240p transcodes added to fill the bandwidth spot from the old 360p ones.

I'll write up a bot that just runs through all the existing videos and resets their .ogv transcodes, a couple at a time.

It's important not to reset too many files at once, or the queue will clog up!

Note webm 240p and 160p have also been added.

Event Timeline

brooke claimed this task.
brooke raised the priority of this task from to Needs Triage.
brooke updated the task description. (Show Details)
brooke added projects: Video, Multimedia.
brooke subscribed.
Jdforrester-WMF moved this task from Untriaged to Next up on the Multimedia board.
Jdforrester-WMF set Security to None.
brooke renamed this task from Run a bot to add 240p, re-run 360p/480p ogv video transcodes to Run a bot to add 240p webm/ogv, re-run 360p/480p ogv video transcodes.May 11 2016, 12:12 AM
brooke updated the task description. (Show Details)

Ok I'm near ready to run this, am testing with audio transcodes before I do anything rash. :)

@brion There are also many (a little over 15k) videos with missing 160.webm transcodes. See https://quarry.wmflabs.org/query/15792.

Currently running:

  • ogg audio (missing)

Next on list:

  • 160p.ogv (missing)
  • 160p.webm (missing)
  • 240p.ogv (missing)
  • 240p.webm (missing)
  • 360p.ogv (all)
  • 480p.ogv (all)

I added a bunch of 160p.ogv transcodes and am waiting to see how long it takes for them to run through. :)

I think I'm going to wait on the rest until I've put an automatic throttle in place... too easy to flood the high-prio queue with requeueTranscodes.php otherwise.

Note per IRC conversation -- will also need to do a generic cleanup pass on non-Commons wikis that have local audio or video uploads. This shouldn't eat too much human time, just needs to be set up and looped.

Currently running throttled version of requeueTranscodes.php to fill in missing 160p.ogv, 160p.webm, 240p.ogv, and 240p.webm transcodes.

There's some spare CPU still, so adding in the 360p.ogv and 480p.ogv re-run jobs, seeing how it goes... (still throttled; currently the high-prio queue is about 5 deep and the low-prio queue is at ~9000, down from ~9800 yesterday)

Change 337230 had a related patch set uploaded (by Brion VIBBER):
Bump up number of queue runners for transcodes

https://gerrit.wikimedia.org/r/337230

Stopping the 360p/480p ogv jobs for now, need to bump up the queue runners (per above patch) to actually use the extra CPU. :)

The 160p/240p.ogv 'missing' jobs are getting closer to done; going ahead and adding the 360p/480p.ogv 'all' jobs back in to the mix.

The queue runner count update (https://gerrit.wikimedia.org/r/337230) didn't get pushed out today due to the puppet SWAT being canceled due to an unrelated flurry of errors in production; will try to poke folks to push it out later.

Change 337230 merged by Giuseppe Lavagetto:
Bump up number of queue runners for transcodes

https://gerrit.wikimedia.org/r/337230

Queue runner count has now been pushed up, so we should get through the rest of the files faster. Might still bump it slightly farther up, I'm seeing usage around 80%, though we're trying to avoid overcommitting CPU either.

It's hard to measure progress exactly without counts in the reqeueueTranscodes.php reporting (I should add that!) but looking at the totals and the progress through the sort order (oggs by name, then webms by name due to the way the index on media type is set up) I think it's about 2/3 through the 160p.ogv missing fill-ins. 240p.ogv and the 160/240p webms are continuing.

Re-running of 360p and 480p ogvs has been churning along, but I'm going to pause those jobs again and see if the low-prio queue falls faster. The throttle on requeueTranscodes.php keeps the queue from growing, but doesn't do much to allow it to shrink...

See grafana graph of the queues -- the response times are ggrreeaatt on the high-prio queue but it's been holding steady at 3-days-ish delay on the low-prio queue the last few days:

Screen Shot 2017-02-15 at 7.31.46 PM.png (924×2 px, 340 KB)

Fill-in jobs for 160p and 240p ogv and webm appear to have completed a while ago. Will re-start the 360p and 480p re-runs after adding one more option to avoid accidentally re-running new transcodes unnecessarily.

If anything got left over, the ogv mid-resolutions no longer need redoing as they've been obsoleted.