Page MenuHomePhabricator

Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty
Closed, ResolvedPublic

Event Timeline

greg created this task.Aug 28 2015, 5:53 PM
greg raised the priority of this task from to Needs Triage.
greg updated the task description. (Show Details)
greg added a subscriber: greg.
Restricted Application added subscribers: Luke081515, Aklapper. · View Herald TranscriptAug 28 2015, 5:53 PM
bd808 claimed this task.Aug 28 2015, 6:03 PM
bd808 triaged this task as High priority.
brion added a subscriber: brion.Aug 28 2015, 6:09 PM

Change 234599 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-videoscaler01 with deployment-tmh01

https://gerrit.wikimedia.org/r/234599

  • Deleted deployment-videoscaler01 instance
  • Created deployment-tmh01 instance using trusty base image
  • Applied beta::deployaccess and role::mediawiki::videoscaler
  • Patched puppet to:
    • Remove deployment-videoscaler01 from /etc/dsh/group/mediawiki-installation
    • Add deployment-tmh01 to /etc/dsh/group/mediawiki-installation
    • Setup jobrunner on deployment-tmh01 to process the transcode job queue

The last bit there (jobrunner setup) had never been done properly for deployment-videoscaler01 and the new jobrunner system. That explains why transcodes were not happening before I started messing around with this.

Change 234681 had a related patch set uploaded (by Brion VIBBER):
Use ffmpeg instead of avconv on labs beta

https://gerrit.wikimedia.org/r/234681

Change 234681 merged by jenkins-bot:
Use ffmpeg instead of avconv on labs beta

https://gerrit.wikimedia.org/r/234681

Change 234699 had a related patch set uploaded (by Brion VIBBER):
Use backported ffmpeg for multimedia transcoding on Trusty

https://gerrit.wikimedia.org/r/234699

Ok this is now mostly-done but there's a problem with the job runner -- it's running more threads than expected, which overloads the scaler VM (which is not as beefy as the production servers). Needs some tweaking.

Just to confirm, the upgrade to HHVM and Trusty is done, yes? If so, let's close this task and you can work out your config/code tweaking as you like :)

brion closed this task as Resolved.Aug 31 2015, 5:53 PM

Yep! I'll file a separate task for the job runner count issue.

bd808 reopened this task as Open.Aug 31 2015, 6:41 PM

I'd like to keep this open until the ops/puppet patches are merged:

word, good point.

Change 234599 merged by Yuvipanda:
beta: Replace deployment-videoscaler01 with deployment-tmh01

https://gerrit.wikimedia.org/r/234599

bd808 added a subscriber: Joe.Sep 9 2015, 12:18 AM

Still waiting on a merge of https://gerrit.wikimedia.org/r/#/c/234699/ (Use backported ffmpeg for multimedia transcoding on Trusty). It was declined for PuppetSWAT® due to interaction with prod imagescalers. Hopefully when @Joe is around he can look at it and merge.

Joe added a comment.Sep 16 2015, 11:05 AM

So, for the jobrunner count issue: that has nothing to do with puppet, but with the jobrunner code directly.

I restarted the jobrunner service and it reports correctly

2015-09-16T11:00:29+0000: Starting job spawner loop(s)...
2015-09-16T11:00:29+0000: Initialized loop 0 with 2 runner(s).

Change 234699 merged by Giuseppe Lavagetto:
Use backported ffmpeg for multimedia transcoding on Trusty

https://gerrit.wikimedia.org/r/234699

Is this done?

Krenair closed this task as Resolved.Sep 29 2015, 12:50 PM

Looks like it:

krenair@deployment-tmh01:~$ php --version
HipHop VM 3.6.5 (rel)
Compiler: 1440023731_994787068
Repo schema: 10ff5b7278f68f074cfdefae238f379500d3148f
Extension API: 20150212
krenair@deployment-tmh01:~$