Page MenuHomePhabricator

Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches
Closed, ResolvedPublic

Description

Rationale on why is at T88436#1013289 and subsequent comments.

Instead of cloning mediawiki/vendor, we should run "composer update". If we are testing extensions, we should create a composer.local.json file that recursively includes those extension's composer.json files if they exist (pending patch).

Related Objects

Event Timeline

Legoktm raised the priority of this task from to Needs Triage.
Legoktm updated the task description. (Show Details)
Legoktm subscribed.

So far:

The Zend PHPUnit jobs are currently running on production slaves...we can't run composer in production so those will need to be moved to labs slaves.

Should we have a separate mediawiki-phpunit-{phpflavor} / mediawiki-phpunit-{phpflavor}-wmf job where one uses composer and the other uses mediawiki/vendor?

Or can we have the job definition automatically vary based on the branch?

Should we have a separate mediawiki-phpunit-{phpflavor} / mediawiki-phpunit-{phpflavor}-wmf job where one uses composer and the other uses mediawiki/vendor?

Or can we have the job definition automatically vary based on the branch?

The job definition can not vary by branch. This is statically interpreted ahead of time and submitted to Jenkins as job.

If the fetching of mediawiki/vendor and composer-update were in a bash script, it could use the environment variable ZUUL_BRANCH at run-time, however in this case it's probably better to change the mediawiki-phpunit-* jobs to use composer, and add a mediawiki-phpunit-*-vendor job that (in Zuul layout) is configured to only run on wmf branches of mediawiki core and extensions (and as main job for mediawiki-vendor itself).

Change 193555 had a related patch set uploaded (by Legoktm):
Use composer for experimental mediawiki-phpunit-{phpflavor}-composer jobs

https://gerrit.wikimedia.org/r/193555

Change 193555 merged by jenkins-bot:
Use composer for experimental mediawiki-phpunit-{phpflavor}-composer jobs

https://gerrit.wikimedia.org/r/193555

Krinkle triaged this task as Medium priority.Mar 2 2015, 1:41 PM

In addition to phpunit, we would need a similar job for qunit, too.

Krinkle renamed this task from Fetch dependencies using composer instead of cloning mediawiki/vendor repository for non-WMF deployment branches to Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches.Mar 25 2015, 1:21 AM

Krinkle and I discussed this a few days ago in #wikimedia-releng. We still need to run composer for wmf branches to fetch dependencies like phpunit. So

  • Non-wmf branches will just run "composer update" to fetch dependencies
  • wmf branches (and mediawiki/vendor patches) will clone mediawiki/vendor, we will copy mediawiki/vendor's composer.json to composer.local.json, run "composer update" (twice?). That will also ensure core's dependencies match vendor's.
  • wmf branches (and mediawiki/vendor patches) will clone mediawiki/vendor, we will copy mediawiki/vendor's composer.json to composer.local.json,

It may be better/easier to just stick in a composer.local.json that in turn includes the mw/vendor composer.json. That way if we end up with some need to include even more things we can use the composer.local.json to do that.

run "composer update" (twice?). That will also ensure core's dependencies match vendor's.

I think it will take two runs. The first run will install our custom composer merge plugin and then the second run will actually use it to squish the two sets of composer dependencies together. We could probably get rid of the need for a second run by adding composer-merge-plugin to mw/vendor.

Change 214569 had a related patch set uploaded (by Legoktm):
composer.json: Downgrade PHPUnit to 3.7.37 (temporarily)

https://gerrit.wikimedia.org/r/214569

Change 214569 merged by jenkins-bot:
composer.json: Downgrade PHPUnit to 3.7.37 (temporarily)

https://gerrit.wikimedia.org/r/214569

This currently does not work, because the jobs using composer right now clone the source to workingspace, but all other jobs that use zuul-cloner use workingspace/src.

Change 232558 had a related patch set uploaded (by JanZerebecki):
[WIP] Change target for clone to src.

https://gerrit.wikimedia.org/r/232558

Change 232558 abandoned by JanZerebecki:
[WIP] Change target for clone to src.

Reason:
To many changes needed for this way. Used https://gerrit.wikimedia.org/r/#/c/237710/2 instead.

https://gerrit.wikimedia.org/r/232558

WARN [Chromium 44.0.2403 (Ubuntu 0.0.0)]: Disconnected (1 times), because no message in 60000 ms.

We need to create a composer version of the jobs in the template extension-gate.

Change 240438 had a related patch set uploaded (by Legoktm):
composer.json: Downgrade PHPUnit to 3.7.37 (temporarily)

https://gerrit.wikimedia.org/r/240438

Change 240438 merged by jenkins-bot:
composer.json: Downgrade PHPUnit to 3.7.37 (temporarily)

https://gerrit.wikimedia.org/r/240438

Change 240951 had a related patch set uploaded (by Legoktm):
composer.json: Downgrade PHPUnit to 3.7.37 (temporarily)

https://gerrit.wikimedia.org/r/240951

Change 240951 merged by jenkins-bot:
composer.json: Downgrade PHPUnit to 3.7.37 (temporarily)

https://gerrit.wikimedia.org/r/240951

Change 247920 had a related patch set uploaded (by Paladox):
Add new extension-unittests-composer template

https://gerrit.wikimedia.org/r/247920

Change 247920 merged by jenkins-bot:
Add new extension-unittests-composer template

https://gerrit.wikimedia.org/r/247920