Page MenuHomePhabricator
Create Task
Maniphest T104747

Convert tmh100[12] to HHVM and trusty
Closed, ResolvedPublic
Actions

Description

They are eqiad videoscaler hosts in WMF production
(codfw videoscalers mw2007 and mw2152 already run trusty+HHVM.)

Details

SubjectRepoBranchLines +/-
videoscaler: reimage and rename tmh1002 as mw1260, upgrade to HAToperations/puppetproduction+9 -14
videoscaler: reimage tmh1001 as mw1259operations/puppetproduction+12 -6
Rename tmh1001 to mw1259 for reimagingoperations/dnsmaster+2 -2
videoscaler: make the HHVM videoscaler the only one runningoperations/puppetproduction+0 -0
Use ffmpeg for video thumbs & transcodingoperations/mediawiki-configmaster+3 -0
Use backported ffmpeg for multimedia transcoding on Trustyoperations/puppetproduction+1 -7
videoscaler: reimage mw1152 as an experimental videoscaleroperations/puppetproduction+9 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krenair created this task.Jul 4 2015, 12:42 AM
Krenair raised the priority of this task from to Needs Triage.
Krenair updated the task description. (Show Details)
Krenair added projects: acl*sre-team, HHVM, TimedMediaHandler.
Krenair subscribed.
Restricted Application added a project: Multimedia. · View Herald TranscriptJul 4 2015, 12:42 AM
Restricted Application added subscribers: Matanya, Aklapper. · View Herald Transcript
Krenair added a parent task: T65899: Upgrade Wikimedia servers to Ubuntu Trusty (14.04).Jul 4 2015, 12:43 AM
Krenair added a parent task: T86081: Complete the use of HHVM over Zend PHP on the Wikimedia cluster.
Krenair updated the task description. (Show Details)Jul 7 2015, 2:45 PM
Krenair set Security to None.
Krenair mentioned this in T105009: videoscaler naming conventions.Jul 7 2015, 2:47 PM
fgiunchedi triaged this task as High priority.Jul 20 2015, 3:20 PM
fgiunchedi subscribed.

looks like we need this to finish migrating to hhvm

ori added a subtask: T107313: Backport ffmpeg 2.7.3 to Trusty.Jul 29 2015, 4:49 PM
ori added a subtask: T69953: ogv transcodes have broken audio tracks with ffmpeg2theora 0.29 (Trusty upstream).Jul 29 2015, 4:52 PM
• brion subscribed.Aug 9 2015, 4:18 PM
fgiunchedi closed subtask T107313: Backport ffmpeg 2.7.3 to Trusty as Resolved.Aug 10 2015, 4:52 PM
fgiunchedi added a subtask: T103335: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie).Aug 10 2015, 4:55 PM
• brion closed subtask T69953: ogv transcodes have broken audio tracks with ffmpeg2theora 0.29 (Trusty upstream) as Resolved.Aug 13 2015, 4:30 PM
Krenair updated the task description. (Show Details)Aug 15 2015, 10:17 AM
Joe claimed this task.Aug 18 2015, 8:00 AM
gerritbot subscribed.Aug 18 2015, 8:02 AM

Change 232242 had a related patch set uploaded (by Giuseppe Lavagetto):
videoscaler: reimage mw1152 as an experimental videoscaler

https://gerrit.wikimedia.org/r/232242

gerritbot added a project: Patch-For-Review.Aug 18 2015, 8:02 AM
gerritbot added a comment.Aug 18 2015, 8:19 AM

Change 232242 merged by Giuseppe Lavagetto:
videoscaler: reimage mw1152 as an experimental videoscaler

https://gerrit.wikimedia.org/r/232242

Joe mentioned this in rOPUP840a434bd403: videoscaler: reimage mw1152 as an experimental videoscaler.Aug 18 2015, 8:20 AM
Joe added a comment.Aug 18 2015, 10:18 AM

mw1152 is successfully reimaged as a videoscaler, and the jobrunner is stopped by default. @brion any suggestions on how to test it specifically?

• brion added a parent task: T109658: Re-enable two-pass .ogv encoding after updating ffmpeg2theora.Aug 19 2015, 11:27 PM
• brion mentioned this in T103335: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie).Aug 24 2015, 4:16 PM

Per IRC discussion: this is probably ready to throw into production, but we ought to test it on beta cluster first like reasonable people. :)

Rough test plan:

  • deployyyyyy (to beta or live)
  • upload some files
  • make sure they render without exploding
hashar subscribed.Aug 24 2015, 4:22 PM
• brion added a comment.Aug 24 2015, 4:29 PM

Ok, sample files to re-run and test:

VP9 source video (currently it won't transcode, but it should work if re-run after reimage):

ogg with lots of motion (should currently work, but should be clearer when re-run after update):

It also looks like there isn't actually anything running transcode jobs on beta yet, at least for en.wikipedia.beta.wmflabs.org ... http://en.wikipedia.beta.wmflabs.org/wiki/Special:TimedMediaHandler <- shows the files queued but no currently running

• brion added a project: Beta-Cluster-Infrastructure.Aug 24 2015, 4:30 PM

Adding beta-cluster project for fixing/updating TMH video scaler job runner for beta cluster...

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptAug 24 2015, 4:30 PM
greg subscribed.Aug 28 2015, 5:54 PM
In T104747#1568015, @brion wrote:

Adding beta-cluster project for fixing/updating TMH video scaler job runner for beta cluster...

Mae a separate task for that: T110707: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty

greg added a subtask: T110707: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty.Aug 28 2015, 5:55 PM
greg removed a project: Beta-Cluster-Infrastructure.Aug 28 2015, 5:58 PM
• brion closed subtask T103335: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) as Resolved.Aug 28 2015, 11:32 PM
gerritbot added a comment.Aug 28 2015, 11:38 PM

Change 234699 had a related patch set uploaded (by Brion VIBBER):
Use backported ffmpeg for multimedia transcoding on Trusty

https://gerrit.wikimedia.org/r/234699

• brion added a comment.Aug 29 2015, 12:52 AM

As noted on T110707 this config looks very close to ready -- it's almost-functional on beta cluster! But we're having trouble controlling the number of threads on the job runner, which implies something's wonky in the config.

• brion closed subtask T110707: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty as Resolved.Aug 31 2015, 5:53 PM
bd808 reopened subtask T110707: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty as Open.Aug 31 2015, 6:41 PM
Jdforrester-WMF moved this task from Untriaged to Doing on the Multimedia board.Sep 3 2015, 4:02 PM
Krenair mentioned this in T86081: Complete the use of HHVM over Zend PHP on the Wikimedia cluster.Sep 14 2015, 12:07 PM
Ricordisamoa subscribed.Sep 14 2015, 12:23 PM
gerritbot added a comment.Sep 16 2015, 4:25 PM

Change 238781 had a related patch set uploaded (by Brion VIBBER):
Use ffmpeg for video thumbs & transcoding

https://gerrit.wikimedia.org/r/238781

gerritbot added a comment.Sep 16 2015, 4:27 PM

Change 234699 merged by Giuseppe Lavagetto:
Use backported ffmpeg for multimedia transcoding on Trusty

https://gerrit.wikimedia.org/r/234699

Joe mentioned this in rOPUP8288b1aca81c: Use backported ffmpeg for multimedia transcoding on Trusty.Sep 16 2015, 4:27 PM
gerritbot added a comment.Sep 16 2015, 4:31 PM

Change 238781 merged by Brion VIBBER:
Use ffmpeg for video thumbs & transcoding

https://gerrit.wikimedia.org/r/238781

• brion mentioned this in rOMWC1a38cb0ed761: Use ffmpeg for video thumbs & transcoding.Sep 16 2015, 4:31 PM
gerritbot added a comment.Sep 16 2015, 5:27 PM

Change 238794 had a related patch set uploaded (by Giuseppe Lavagetto):
videoscaler: make the HHVM videoscaler the only one running

https://gerrit.wikimedia.org/r/238794

gerritbot added a comment.Sep 16 2015, 5:30 PM

Change 238794 merged by Giuseppe Lavagetto:
videoscaler: make the HHVM videoscaler the only one running

https://gerrit.wikimedia.org/r/238794

Joe mentioned this in rOPUP1d5df79555a1: videoscaler: make the HHVM videoscaler the only one running.Sep 16 2015, 5:30 PM
Joe added a comment.Sep 16 2015, 5:36 PM

As of now, all tmh jobs are being handled by the newer trusty-based host.

To change this it's sufficient to revert https://gerrit.wikimedia.org/r/238794

gerritbot added a comment.Sep 17 2015, 11:28 AM

Change 239078 had a related patch set uploaded (by Giuseppe Lavagetto):
videoscaler: reimage tmh1001 as mw1259

https://gerrit.wikimedia.org/r/239078

gerritbot added a comment.Sep 17 2015, 1:54 PM

Change 239103 had a related patch set uploaded (by Giuseppe Lavagetto):
Rename tmh1001 to mw1259 for reimaging

https://gerrit.wikimedia.org/r/239103

gerritbot added a comment.Sep 17 2015, 2:05 PM

Change 239103 merged by Giuseppe Lavagetto:
Rename tmh1001 to mw1259 for reimaging

https://gerrit.wikimedia.org/r/239103

Joe mentioned this in rODNS2bb2926326ec: Rename tmh1001 to mw1259 for reimaging.Sep 17 2015, 2:06 PM
gerritbot added a comment.Sep 17 2015, 2:07 PM

Change 239078 merged by Giuseppe Lavagetto:
videoscaler: reimage tmh1001 as mw1259

https://gerrit.wikimedia.org/r/239078

Joe mentioned this in rOPUP1b7e57beff1e: videoscaler: reimage tmh1001 as mw1259.Sep 17 2015, 2:07 PM
Jdforrester-WMF removed a project: Multimedia.Sep 21 2015, 4:01 PM
• brion mentioned this in T113284: Frequent job timeouts on HHVM video scalers.Sep 21 2015, 7:57 PM
• brion added a subtask: T113284: Frequent job timeouts on HHVM video scalers.
Joe closed subtask T113284: Frequent job timeouts on HHVM video scalers as Resolved.Sep 29 2015, 12:46 PM
Krenair closed subtask T110707: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty as Resolved.Sep 29 2015, 12:50 PM
gerritbot added a comment.Sep 29 2015, 1:44 PM

Change 242125 had a related patch set uploaded (by Giuseppe Lavagetto):
videoscaler: reimage and rename tmh1002 as mw1260, upgrade to HAT

https://gerrit.wikimedia.org/r/242125

gerritbot added a comment.Sep 29 2015, 2:06 PM

Change 242125 merged by Giuseppe Lavagetto:
videoscaler: reimage and rename tmh1002 as mw1260, upgrade to HAT

https://gerrit.wikimedia.org/r/242125

Joe mentioned this in rOPUPd0b317c89f8b: videoscaler: reimage and rename tmh1002 as mw1260, upgrade to HAT.Sep 29 2015, 2:07 PM
Joe closed this task as Resolved.Sep 29 2015, 4:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL