Page MenuHomePhabricator

Come up with non sucky solution for running "composer test" on repos that have vendor/ checked in
Closed, ResolvedPublic

Description

We're migrating towards running tests with "composer test". This requires that phpunit/lint/phpcs are specified as dev dependencies in composer.json. Some of our repos have their dependencies checked in for deployment purposes:

  • operations/mediawiki-config
  • wikimedia/ieg-review
  • wikimedia/wikimania-scholarships

To run tests for those repos, we'd have to do something like composer update --dev, composer test, and then composer update --no-dev to get rid of the dev dependencies when committing. That sucks. Is there a better way to do this?

Event Timeline

Legoktm created this task.Mar 13 2015, 6:08 AM
Legoktm raised the priority of this task from to Needs Triage.
Legoktm updated the task description. (Show Details)
Legoktm added subscribers: Legoktm, bd808.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2015, 6:08 AM
bd808 added a comment.Mar 13 2015, 2:55 PM

One approach would be to split the deployable vendor repo from the project repo and only combine them on a deployment branch or via Trebuchet as appropriate. The downsides of this are more complex git repo management for the production and possibility of drift between the composer.json in the main project vs the composer.json in the vendor repo, but the upside is easier local development.

Perhaps we can change those repos to not check in vendor/? MediaWiki core and extensions don't do that either.

For the main test job we'd let Jenkins fetch dependencies (dependencies and dev, via composer test), and for the wmf branch (or in case of mediawiki-config etc.) we'd use mediawiki/vendor.

bd808 added a comment.Apr 14 2015, 4:15 PM

Perhaps we can change those repos to not check in vendor/? MediaWiki core and extensions don't do that either.
For the main test job we'd let Jenkins fetch dependencies (dependencies and dev, via composer test), and for the wmf branch (or in case of mediawiki-config etc.) we'd use mediawiki/vendor.

This is basically what I suggested in T92605#1116654. The downside is more complex deployment. For ieg-review and scholarships this can be handled by Trebuchet pretty easily.

For mediawiki-config I'm not exactly sure what we would do. We could use an independent git clone I suppose. It doesn't currently seem likely that we will be adding more libraries to the bootstrapping code for the WMF cluster.

For mediawiki-config I'm not exactly sure what we would do. We could use an independent git clone I suppose. It doesn't currently seem likely that we will be adding more libraries to the bootstrapping code for the WMF cluster.

Or a submodule? (Like for mediawiki-core wmf branches.) It'd be less error prone and more explicit. A bit more maintenance heavy, too. But as you say, they shouldn't change very often.

Tgr added a subscriber: Tgr.Aug 26 2015, 4:54 AM

Just put dev dependencies in gitignore? One could probably even write a composer plugin that does that automatically, if it is worth the effort.

hashar triaged this task as Normal priority.Oct 6 2015, 11:25 AM
hashar set Security to None.
bd808 added a comment.Oct 7 2015, 8:07 PM
In T92605#1574097, @Tgr wrote:

Just put dev dependencies in gitignore? One could probably even write a composer plugin that does that automatically, if it is worth the effort.

composer.lock and various files in vendor/composer will change when the installed libs change so a simple gitignore won't quite cover it.

If the dependencies are not pinned to exact versions (which is often the case for projects that version composer.lock) then the composer update && composer update --no-dev dance won't be guaranteed to work properly either as both runs of update could choose to install non-dev dependencies than the ones that are in the versioned composer.lock.

What I have been doing locally for a couple projects with this problem is composer update jakub-onderka/php-parallel-lint phpunit/phpunit ... listing each package in the require-dev section and then after I'm done running tests resetting to the committed state with git checkout -- composer.lock and any other files that were dirtied.

Change 330842 had a related patch set uploaded (by Krinkle):
Add fetch-composer-dev.sh script (derived from mw-fetch-composer-dev.sh)

https://gerrit.wikimedia.org/r/330842

Change 330843 had a related patch set uploaded (by Krinkle):
Add operations-mw-config-composer-jessie job

https://gerrit.wikimedia.org/r/330843

Krinkle claimed this task.Jan 6 2017, 4:48 AM

Change 330842 merged by jenkins-bot:
Add fetch-composer-dev.sh script (derived from mw-fetch-composer-dev.sh)

https://gerrit.wikimedia.org/r/330842

Change 330843 merged by jenkins-bot:
Add operations-mw-config-composer-jessie job

https://gerrit.wikimedia.org/r/330843

Krinkle closed this task as Resolved.Jan 10 2017, 11:05 PM