Page MenuHomePhabricator
Create Task
Maniphest T186145

Migrate operations-mw-config-composer-hhvm-jessie to Docker
Closed, ResolvedPublic
Actions

Description

We should migrate the job operations-mw-config-composer-hhvm-jessie to a Docker based one.

operations/mediawiki-config has composer.lock and vendor directory committed in. Though we would still need to install the development dependencies.

Giving it a try with a container:

cd ~/projects/operations/mediawiki-config
hashar@pico:~/projects/operations/mediawiki-config(masteru=)$ docker run  --rm -it -v "$(pwd)":/src docker-registry.wikimedia.org/releng/composer-test-hhvm:0.2.1
+ cd /src
+ composer --ansi validate --no-check-publish
./composer.json is valid, but with a few warnings
See https://getcomposer.org/doc/04-schema.md for details on the schema
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.
require.mongofill/mongofill : unbound version constraints (dev-master) should be avoided
+ composer install --no-progress --prefer-dist
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for doctrine/instantiator 1.1.0 -> satisfiable by doctrine/instantiator[1.1.0].
    - doctrine/instantiator 1.1.0 requires php ^7.1 -> your HHVM version does not satisfy that requirement.
  Problem 2
    - Installation request for phpdocumentor/reflection-docblock 4.1.1 -> satisfiable by phpdocumentor/reflection-docblock[4.1.1].
    - phpdocumentor/reflection-docblock 4.1.1 requires php ^7.0 -> your HHVM version does not satisfy that requirement.
  Problem 3
    - doctrine/instantiator 1.1.0 requires php ^7.1 -> your HHVM version does not satisfy that requirement.
    - phpunit/phpunit-mock-objects 2.3.8 requires doctrine/instantiator ^1.0.2 -> satisfiable by doctrine/instantiator[1.1.0].
    - Installation request for phpunit/phpunit-mock-objects 2.3.8 -> satisfiable by phpunit/phpunit-mock-objects[2.3.8].

The Jenkins job is very specific to that repo, it is in integration/config:

jjb/operations-misc.yaml
# Runs 'composer validate' and 'composer test', for the
# operations/mediawiki-config repo. Uses fetch-composer-dev to install require-dev
# packages while preserving non-dev packages from vendor in Git. (T85947)
# Avoid generic jobs to ensure it stays out of the
# MediaWiki gate-and-submit queue. (T101908)
- job:
    name: 'operations-mw-config-composer-hhvm-jessie'
    node: ci-jessie-wikimedia
    defaults: use-remoteonly-zuul
    concurrent: true
    triggers:
     - zuul
    builders:
     - castor-load
     - composer-validate:
         dir: '.'
     - fetch-composer-dev
     - composer-test:
         dir: '.'
    publishers:
     - castor-save

We need the composer container to somehow run fetch-composer-dev (that is a slave script in integration/jenkins) and skip composer install entirely.

I am not sure how it works on the Nodepool instance. The PHP version should be set to hhvm.

Details

SubjectRepoBranchLines +/-
Switch mw-config composer job to Dockerintegration/configmaster+3 -27
Experimental mw-config composer job on Dockerintegration/configmaster+28 -0
Add support to pass options to 'docker run'integration/configmaster+3 -0
docker: script to only install composer dev dependenciesintegration/configmaster+91 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Jan 31 2018, 7:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2018, 7:53 PM
hashar added a project: Release-Engineering-Team (Kanban).Jan 31 2018, 7:54 PM
Addshore subscribed.Feb 1 2018, 3:26 PM

+999 for this and mediawiki-config no longer having to wait for nodepool!

  • doctrine/instantiator 1.1.0 requires php ^7.1 -> your HHVM version does not satisfy that requirement.

Lets run it in a hhvm container instead of a php container?

Or we can ignore platform deps (php version is one of them if I am not mistaken)

hashar added a comment.Feb 1 2018, 7:06 PM

It is an HHVM container :-) I am not sure yet why it works on a Nodepool Jessie instance though. So to be investigated.

Addshore added a comment.Feb 1 2018, 8:22 PM
In T186145#3938606, @hashar wrote:

It is an HHVM container :-) I am not sure yet why it works on a Nodepool Jessie instance though. So to be investigated.

I meant php7, but i guess we want to be running this stuff on hhvm as that's what we run in prod..

hashar added a subtask: T181938: Swap node for jq in mw-fetch-composer-dev.sh.Feb 15 2018, 3:20 PM
hashar added a comment.Feb 15 2018, 3:25 PM

The dev dependencies are not included in multiversion/vendor though at least they are in composer.lock. To get them installed the job uses the jjb macro fetch-composer-dev:

jjb/macro.yaml
- builder:
    name: fetch-composer-dev
    builders:
     - shell: |
        #!/bin/bash -eu
        # Load require-dev packages on top of vendor (T92605, T112895)
        set -xo pipefail
        # Set --ignore-platform-reqs to prevent composer from evaluating unrelated dependencies
        # already present in composer.json (needed for operations/mediawiki-config due to
        # https://github.com/perftools/xhgui/issues/192).
        export HTTP_PROXY_REQUEST_FULLURI=false
        export HTTPS_PROXY_REQUEST_FULLURI=false
        node /srv/deployment/integration/slave-scripts/tools/composer-dev-args.js "$(pwd)/composer.json" | xargs --verbose composer require --dev --ansi --no-progress --ignore-platform-reqs --prefer-dist -v

That is very similar with one of first lines of bin/mw-fetch-composer-dev.sh:

# Load require-dev packages on top of mediawiki/vendor (T112895)
# Once jq 1.4 is available, the following may be used instead:
# > jq -r '.["require-dev"]|to_entries|map([.key,.value])[]|join("=")' ../composer.json
set -o pipefail
node $(dirname $0)/../tools/composer-dev-args.js "$MW_INSTALL_PATH/composer.json" | xargs --verbose composer require --dev --ansi --no-progress --prefer-dist -v

I would rather avoid adding node to the releng/composer-test-hhvm container, but I guess we can add jq which is partly T181938

gerritbot subscribed.Feb 15 2018, 3:37 PM

Change 410950 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate mw-config composer job to Docker

https://gerrit.wikimedia.org/r/410950

gerritbot added a project: Patch-For-Review.Feb 15 2018, 3:37 PM
hashar mentioned this in T181938: Swap node for jq in mw-fetch-composer-dev.sh.Feb 15 2018, 3:37 PM
gerritbot added a comment.Feb 15 2018, 4:02 PM

Change 410990 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: add jq to composer images

https://gerrit.wikimedia.org/r/410990

hashar claimed this task.Feb 15 2018, 4:52 PM
hashar triaged this task as High priority.
hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
hashar added a comment.Feb 15 2018, 8:30 PM

With the containers of https://gerrit.wikimedia.org/r/#/c/410990/ I got the test suite to run properly with:

docker run --rm -it --entrypoint=/srv/composer/composer-install-dev-only -v /home/hashar/projects/operations/mediawiki-config:/src docker-registry.wikimedia.org/releng/composer-test-hhvm:0.2.2 /src --profile --ignore-platform-reqs
docker run --rm -it -v ~/projects/operations/mediawiki-config:/src docker-registry.wikimedia.org/releng/composer-hhvm:0.2.1 --ansi --working-dir=/src test
gerritbot added a comment.Feb 15 2018, 8:32 PM

Change 410990 merged by jenkins-bot:
[integration/config@master] docker: script to only install composer dev dependencies

https://gerrit.wikimedia.org/r/410990

gerritbot added a comment.Feb 15 2018, 9:13 PM

Change 411098 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Add support to pass options to 'docker run'

https://gerrit.wikimedia.org/r/411098

hashar added a comment.Feb 15 2018, 9:30 PM

I forged a job that seems to pass all fine and do exactly what we want:
https://integration.wikimedia.org/ci/job/operations-mw-config-composer-test-docker/

hashar added a comment.Feb 15 2018, 10:47 PM
In T186145#3939064, @Addshore wrote:
In T186145#3938606, @hashar wrote:

It is an HHVM container :-) I am not sure yet why it works on a Nodepool Jessie instance though. So to be investigated.

I meant php7, but i guess we want to be running this stuff on hhvm as that's what we run in prod..

I went with the releng/composer-hhvm container since the Nodepool job has PHP_BIN=hhvm and that is the main use case for production right now. I guess we can then add a second job that is based on the releng/composer-php7 container.

gerritbot added a comment.Feb 16 2018, 11:07 AM

Change 411098 merged by jenkins-bot:
[integration/config@master] Add support to pass options to 'docker run'

https://gerrit.wikimedia.org/r/411098

gerritbot added a comment.Feb 16 2018, 11:09 AM

Change 410950 merged by jenkins-bot:
[integration/config@master] Experimental mw-config composer job on Docker

https://gerrit.wikimedia.org/r/410950

gerritbot added a comment.Feb 16 2018, 11:26 AM

Change 411206 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Switch mw-config composer job to Docker

https://gerrit.wikimedia.org/r/411206

gerritbot added a comment.Feb 16 2018, 11:29 AM

Change 411206 merged by jenkins-bot:
[integration/config@master] Switch mw-config composer job to Docker

https://gerrit.wikimedia.org/r/411206

hashar closed this task as Resolved.Feb 16 2018, 11:29 AM

I have switched operations/mediawiki-config job for composer to Docker via https://gerrit.wikimedia.org/r/#/c/411206/

Reedy closed subtask T181938: Swap node for jq in mw-fetch-composer-dev.sh as Resolved.Feb 20 2018, 10:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL