Page MenuHomePhabricator

Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty
Closed, ResolvedPublic

Event Timeline

greg created this task.Aug 28 2015, 5:53 PM
greg raised the priority of this task from to Needs Triage.
greg updated the task description. (Show Details)
greg added a subscriber: greg.
Restricted Application added subscribers: Luke081515, Aklapper. · View Herald TranscriptAug 28 2015, 5:53 PM
bd808 claimed this task.Aug 28 2015, 6:03 PM
bd808 triaged this task as High priority.
brion added a subscriber: brion.Aug 28 2015, 6:09 PM

Change 234599 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-videoscaler01 with deployment-tmh01

https://gerrit.wikimedia.org/r/234599

  • Deleted deployment-videoscaler01 instance
  • Created deployment-tmh01 instance using trusty base image
  • Applied beta::deployaccess and role::mediawiki::videoscaler
  • Patched puppet to:
    • Remove deployment-videoscaler01 from /etc/dsh/group/mediawiki-installation
    • Add deployment-tmh01 to /etc/dsh/group/mediawiki-installation
    • Setup jobrunner on deployment-tmh01 to process the transcode job queue

The last bit there (jobrunner setup) had never been done properly for deployment-videoscaler01 and the new jobrunner system. That explains why transcodes were not happening before I started messing around with this.

Change 234681 had a related patch set uploaded (by Brion VIBBER):
Use ffmpeg instead of avconv on labs beta

https://gerrit.wikimedia.org/r/234681

Change 234681 merged by jenkins-bot:
Use ffmpeg instead of avconv on labs beta

https://gerrit.wikimedia.org/r/234681

Change 234699 had a related patch set uploaded (by Brion VIBBER):
Use backported ffmpeg for multimedia transcoding on Trusty

https://gerrit.wikimedia.org/r/234699

Ok this is now mostly-done but there's a problem with the job runner -- it's running more threads than expected, which overloads the scaler VM (which is not as beefy as the production servers). Needs some tweaking.

greg added a comment.Aug 31 2015, 5:35 PM

Just to confirm, the upgrade to HHVM and Trusty is done, yes? If so, let's close this task and you can work out your config/code tweaking as you like :)

brion closed this task as Resolved.Aug 31 2015, 5:53 PM

Yep! I'll file a separate task for the job runner count issue.

bd808 reopened this task as Open.Aug 31 2015, 6:41 PM

I'd like to keep this open until the ops/puppet patches are merged:

greg added a comment.Aug 31 2015, 6:58 PM

word, good point.

Change 234599 merged by Yuvipanda:
beta: Replace deployment-videoscaler01 with deployment-tmh01

https://gerrit.wikimedia.org/r/234599

bd808 added a subscriber: Joe.Sep 9 2015, 12:18 AM

Still waiting on a merge of https://gerrit.wikimedia.org/r/#/c/234699/ (Use backported ffmpeg for multimedia transcoding on Trusty). It was declined for PuppetSWAT® due to interaction with prod imagescalers. Hopefully when @Joe is around he can look at it and merge.

Joe added a comment.Sep 16 2015, 11:05 AM

So, for the jobrunner count issue: that has nothing to do with puppet, but with the jobrunner code directly.

I restarted the jobrunner service and it reports correctly

2015-09-16T11:00:29+0000: Starting job spawner loop(s)...
2015-09-16T11:00:29+0000: Initialized loop 0 with 2 runner(s).

Change 234699 merged by Giuseppe Lavagetto:
Use backported ffmpeg for multimedia transcoding on Trusty

https://gerrit.wikimedia.org/r/234699

Is this done?

Krenair closed this task as Resolved.Sep 29 2015, 12:50 PM

Looks like it:

krenair@deployment-tmh01:~$ php --version
HipHop VM 3.6.5 (rel)
Compiler: 1440023731_994787068
Repo schema: 10ff5b7278f68f074cfdefae238f379500d3148f
Extension API: 20150212
krenair@deployment-tmh01:~$