Page MenuHomePhabricator

[OPS] Jenkins: Slaves running Ubuntu Trusty should have hhvm installed
Closed, ResolvedPublic

Description

The hhvm package is no more being installed on the Trusty continuous integration slaves, as an example integration-slave1009.eqiad.wmflabs has been started recently and is lacking hhvm.

We set up the slaves using the role class role::ci::slave::labs which loads contint::packages and ::mediawiki::packages . Maybe that later class included hhvm at some point.

We need a class in puppet to install hhvm, optionally ensure => latest and ideally with whatever hhvm configuration file that will be suitable in the CI context.


Version: wmf-deployment
Severity: normal

Details

Reference
bz73356

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:54 AM
bzimport set Reference to bz73356.
bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.Nov 13 2014, 10:49 AM

Timo, I have unpooled integration-slave1009 since it is missing hhvm and left a note pointing to this bug. https://integration.wikimedia.org/ci/computer/integration-slave1009/

(In reply to Antoine "hashar" Musso (WMF) from comment #1)

Timo, I have unpooled integration-slave1009 since it is missing hhvm and
left a note pointing to this bug.
https://integration.wikimedia.org/ci/computer/integration-slave1009/

I'm not sure that's needed.

While integration-slave1006, 1007 and 1008 do have hhvm installed. None of them run hhvm under the 'php' command. And we don't use hhvm anywhere yet in Jenkins.

Jobs using PHP are assigned to Precise slaves, not Trusty. Because that comes with PHP 5.4/ PHP 5.5 which we don't want to use. Once we have hvvm installed and php aliased to hhvm, we can start creating hhvm jobs but afaik integration-slave1009 is equipped to be pooled. The others having hhvm is a coincidence at this point, it's not part of the manifest yet afaik.

hashar added a project: acl*sre-team.
hashar set Security to None.

Moved to Operations , need to figure out an entry point in puppet manifests to easily install hhvm on the contint slaves.

hashar renamed this task from Jenkins: Slaves running Ubuntu Trusty should have hhvm installed to [OPS] Jenkins: Slaves running Ubuntu Trusty should have hhvm installed.Nov 24 2014, 3:45 PM
hashar added a comment.Dec 1 2014, 5:07 PM

I have poked the ops list to get help on installing hhvm and properly configure it on Jenkins slaves.

greg triaged this task as High priority.Dec 8 2014, 6:25 PM
mmodell added a subscriber: mmodell.Dec 9 2014, 5:32 PM
hashar added a comment.Dec 9 2014, 9:29 PM

On top of my head and from a thread I posted on the internal ops list. That is a bit messy but I think it covers the needs.

We want to use HHVM for the PHPUnit jobs and the qunit.

The Zend PHP jobs are tied to Precise instances which comes with PHP 5.3.x. HHVM PHP jobs need to be tied to Trusty instances.

The HHVM package can be installed via the puppet class ::hhvm. The settings there need to be investigated and tweaked for the CI context.

@hashar wrote:
We have a very [lame script] to run PHPUnit which invokes:
hhvm -vEval.Jit=1 \

-vEval.PerfPidMap=false \
-vDebug.CoreDumpReportDirectory="$LOG_DIR" \
-vRepo.Central.Path="$WORKSPACE/hhvm.hhbc.sqlite" \
--php phpunit.php \
--with-phpunitdir "$PHPUNIT_DIR" \
--conf "$MW_INSTALL_PATH/LocalSettings.php" \
--log-junit $JUNIT_DEST

[lame script]
http://git.wikimedia.org/blob/integration%2Fjenkins/master/bin%2Fmw-run-phpunit-hhvm.sh#L42

A few more questions I had:


  • can we override HHVM parameters via env variable? Repo.Central.Path comes to mind since we don't want different builds/code to use the same repo.

Ori replied:

If you leave the repo path ini setting unset, HHVM will look for HHVM_REPO_CENTRAL_PATH in the environment. Doing it via the command-line (as you did previously) is fine too.

Giuseppe recommended to use a shell wrapper around HHVM to define the settings.


  • I had to set Eval.Jit=1, otherwise the PHPUnit tests were damn too

slow. The cli defaults in puppet have it false though.

Ori replied:

The JIT is not beneficial for short-running invocations. Since the unit tests take a while to run, it's not surprising that they benefit from the JIT. It's fine to have it on.


  • The qunit tests are running using apache mod_php5 (we include the

mediawiki class which includes apache::mod::php5) and points to a vhost
localhost:9413. Is there a straight forward way to add the fastcgi
frontend for hhvm?

Ori replied:

Yep. Just include ::apache::mod::proxy_fcgi. A minimal (but functional) Apache 2.4 vhost for MediaWiki on HHVM looks like this: https://gist.github.com/atdt/8dcd1ebcb442efd2585f


We will have to figure out how to vary settings on a per build basis, since the various HHVM caches should probably not be shared. The Repo.Central.Path parameter can be overridden with env variable HHVM_REPO_CENTRAL_PATH.

On Ubuntu /usr/bin/php is managed by the Debian alternatives system (Zend being /usr/bin/php5 and HHVM /usr/bin/hhvm). The Jenkins jobs just use 'php' but I noticed the alternative can be borked and point to either php5 or hhvm :/

We will probably want a php shell script wrapper inserted in the PATH environment, this way we can have Jenkins switch easily between the flavors and still reference 'php'. Something like:

#!/bin/bash -e
HHVM_OPTS="foo"

case $PHP_FLAVOR in
    ( 'hhvm' ) exec /usr/bin/hhvm "$HHVM_OPTS" "$@" ;;
    ( 'zend' ) exec /usr/bin/php5 "$@" ;;
    ( * ) 
        echo "\$PHP_FLAVOR unrecognized ( hhvm | zend )"
        exit 1 ;;
esac

Finally have to figure out the Jenkins Job Builder changes to generate jobs for both flavors and without copy pasting which is exactly what it is meant for. Each job templates would just set a PHP_FLAVOR parameter which is populated in the build environment and then used by the php wrapper.

hashar claimed this task.Dec 9 2014, 9:30 PM

Will fill a bunch of subtasks eventually so people can pick parts.

We will probably want a php shell script wrapper inserted in the PATH environment, this way we can have Jenkins switch easily between the flavors and still reference 'php'. Something like:

#!/bin/bash -e
HHVM_OPTS="foo"
case $PHP_FLAVOR in
    ( 'hhvm' ) exec /usr/bin/hhvm "$HHVM_OPTS" "$@" ;;
    ( 'zend' ) exec /usr/bin/php5 "$@" ;;
    ( * ) 
        echo "\$PHP_FLAVOR unrecognized ( hhvm | zend )"
        exit 1 ;;
esac

If you want to go this route, use https://github.com/CHH/phpenv just like Travis-CI does. It's a variant of rbenv that setup up a virtual tree for switching PHP interpreter versions from the CLI or via a magic .php-version file in a directory. I use it on my laptop and in several testing VMs to make testing things in multiple PHP interpreters easy. In a jenkins job you can then parametrize the PHP version and switch versions easily.

@hashar wrote:
We have a very [lame script] to run PHPUnit which invokes:
hhvm -vEval.Jit=1 \

-vEval.PerfPidMap=false \
-vDebug.CoreDumpReportDirectory="$LOG_DIR" \
-vRepo.Central.Path="$WORKSPACE/hhvm.hhbc.sqlite" \
--php phpunit.php \
--with-phpunitdir "$PHPUNIT_DIR" \
--conf "$MW_INSTALL_PATH/LocalSettings.php" \
--log-junit $JUNIT_DEST

[lame script]
http://git.wikimedia.org/blob/integration%2Fjenkins/master/bin%2Fmw-run-phpunit-hhvm.sh#L42

This looks about right to me honestly. You may find a few more -vWHATEVER flags you want to apply but it should work fine. What's the major problem with this approach?

The easiest route to HHVM testing would seem to me to be to spin up Ubuntu 14.04 slaves, install HHVM and run tests there via Jenkins tags that pin the HHVM jobs to 14.04 slaves. I can see the utility in a nicer multiple PHP interpreter setup as well, but it seems like a big bite to take in order to accomplish the near term goal.

Most of the puppet config for HHVM in operations/puppet is runtime configuration that can be handled by the tests themselves if you use a wrapper script like the one you showed. If you don't use a wrapper script and expect php foo.php to run HHVM on some servers and PHP 5.x on others you will still need to pin jobs to particular OS versions on the slaves or delve into custom PHP builds to get older PHP versions tested on newer slaves.

100% of the WMF app and api servers are now running HHVM. I imagine that the production jobrunners and imagescalers will follow sooner rather than later. All of beta is running HHVM. We should be treating HHVM as a first class citizen for testing.

The easiest route to HHVM testing would seem to me to be to spin up Ubuntu 14.04 slaves, install HHVM and run tests there via Jenkins tags that pin the HHVM jobs to 14.04 slaves. I can see the utility in a nicer multiple PHP interpreter setup as well, but it seems like a big bite to take in order to accomplish the near term goal.

php points to the Zend PHP by default, but we can set the Debian alternative to hhvm on CI slaves. We have a bunch of scripts invoking php directly (ex: shell shebangs, php maintenance update.php), so we would need sane defaults settings in the HHVM configuration files and settings that needs to vary on a per build basis to be passed as env variables (ex: HHVM_REPO_CENTRAL_PATH="$WORKSPACE/hhvm.hhbc.sqlite")

Most of the puppet config for HHVM in operations/puppet is runtime configuration that can be handled by the tests themselves if you use a wrapper script like the one you showed. If you don't use a wrapper script and expect php foo.php to run HHVM on some servers and PHP 5.x on others you will still need to pin jobs to particular OS versions on the slaves or delve into custom PHP builds to get older PHP versions tested on newer slaves.

We pin jobs already:

  • Zend PHP 5.3.x to Precise (since Trusty has 5.5 and we want to make sure we are 5.3 compatible)
  • HHVM jobs are pined to Trusty since Precise does not have it. Then Trusty defaults php to Zend PHP.

100% of the WMF app and api servers are now running HHVM. I imagine that the production jobrunners and imagescalers will follow sooner rather than later. All of beta is running HHVM. We should be treating HHVM as a first class citizen for testing.

Yeah finally. That has been acknowledged by Greg and raised during the RelEng weekly checkin. Now a high priority.

Change 178806 had a related patch set uploaded (by Hashar):
contint: provision hhvm on CI slaves

https://gerrit.wikimedia.org/r/178806

Patch-For-Review

I have applied the puppet patch https://gerrit.wikimedia.org/r/178806 on the continuous integration puppetmaster, so we have hhvm installed and more or less setup for cgi scripts. Still have to figure out the proper configuration for fcgi runs (i.e. for the qunit jobs).

hashar closed this task as Resolved.Jan 8 2015, 10:08 AM
hashar moved this task from In-progress to Done on the Continuous-Integration-Infrastructure board.

Patch https://gerrit.wikimedia.org/r/#/c/178806/ is still pending review but otherwise has been already deployed. We have Jenkins jobs running PHPUnit under hhvm for mediawiki/core already.

So not much left to do beside getting the patch polished up and merged in operations/puppet.git

Change 178806 merged by Faidon Liambotis:
contint: provision hhvm on CI slaves

https://gerrit.wikimedia.org/r/178806