Handle composer require-dev together with mediawiki-vendor without overwriting the things that are already provided by mediawiki-vendor. Or find a different solution for how to have a phpunit around to run.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | |
|---|---|---|---|---|
| · · · | ||||
| Resolved | None | T90303 Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches | ||
| Open | None | T115032 phpcs doesn't work with mediawiki/vendor repo, while phpunit fails with composer-managed vendor. | ||
| Resolved | Krinkle | T112867 make CI for extensions able to run PHPUnit from composer instead of a system wide installation | ||
| Resolved | Krinkle | T99982 Upgrade PHPUnit to 4.0+ | ||
| Resolved | Krinkle | T92605 Come up with non sucky solution for running "composer test" on repos that have vendor/ checked in | ||
| Resolved | Krinkle | T112895 Support installing composer require-dev packages together with mediawiki/vendor | ||
| · · · |
Event Timeline
Comment Actions
I was thinking of: clone mediawiki/vendor, add use composer-merge-plugin to include mediawiki/core's composer.json, and then run composer install, which should fetch new dev deps, but not bump anything that is already installed from vendor.
Comment Actions
If one updated something in core but not vendor that would still lead to an update instead of a failed job. Doing that while merging the composer.json from vendor might work, as long as that composer.json only ever contains exact version requirements. If we stopped using exact versions there, we could still generate one from the composer.lock in vendor.
Comment Actions
I was wondering the if the same thing would work. We could even extend composer-merge-plugin to allow configuration of what sections are allowed to be merged and setup mediawiki/vendor to only allow require-dev rather than everything.
Otherwise I think we would want the composer.lock from mediawiki/vendor to be used with the composer.json from mediawiki/core and ... something ... that would let us only bring in the require-dev libs and dependencies.
My version of composer (which I think is the latest) doesn't have an --only-dev flag for composer update.
The only way I know to pick and choose updates is with the composer update vendor/package syntax.
Comment Actions
This is blocking various things:
- Updating MediaWiki core to use PHPUnit specified in composer.json.
- Which in turn blocks out ability to improve PHPUnit performance (newer versions are much more optimised) as well as newer PHPunit features and various bug fixes in its mocking library.
- Improving MediaWIki core unit test performance by using vfsStream (virtual file system for PHP) because we can't specify it as a require-dev dependency since it isn't being installed.
And various other things that make use of composer for tooling around MediaWiki core.
Comment Actions
This might actually be pretty easy to script for Jenkins jobs:
- Clone mediawiki/core
- Clone mediawiki/vendor
- Parse the "require-dev" bits out of core/composer.json
- Install those packages explictly
Here's a POC script that shows one way to do that:
#!/usr/bin/env bash set -xe git clone --depth 1 https://github.com/wikimedia/mediawiki.git cd mediawiki git clone --depth 1 https://github.com/wikimedia/mediawiki-vendor.git vendor cd vendor composer show --installed > from-vendor jq -r '.["require-dev"]|to_entries|map([.key,.value])[]|join("=")' ../composer.json | xargs composer require composer show --installed > after-require diff -uw from-vendor after-require
The result of the diff check at the end for me locally:
--- from-vendor 2015-12-15 16:53:48.000000000 -0700 +++ after-require 2015-12-15 16:53:55.000000000 -0700 @@ -1,16 +1,28 @@ composer/semver 1.2.0 Semver library that offers utiliti... cssjanus/cssjanus v1.1.1 Convert CSS stylesheets between le... firebase/php-jwt v3.0.0 A simple library to encode and dec... +jakub-onderka/php-parallel-lint v0.9 This tool check syntax of PHP file... +justinrainbow/json-schema 1.5.0 A library to validate a json schema. kzykhys/pygments v1.0.0 A Thin Wrapper for the Python Pygm... liuggio/statsd-php-client v1.0.18 Statsd (Object Oriented) client li... mediawiki/at-ease v1.1.0 Safe replacement to @ for suppress... +mediawiki/mediawiki-codesniffer v0.4.0 MediaWiki CodeSniffer Standards monolog/monolog 1.17.2 Sends your logs to files, sockets,... nmred/kafka-php v0.1.4 Kafka client for php oojs/oojs-ui v0.14.1 Provides library of common widgets... oyejorge/less.php v1.7.0.9 PHP port of the Javascript version... +phpunit/php-code-coverage 1.2.18 Library that provides collection, ... +phpunit/php-file-iterator 1.4.1 FilterIterator implementation that... +phpunit/php-text-template 1.2.1 Simple template engine. +phpunit/php-timer 1.0.7 Utility class for timing +phpunit/php-token-stream 1.2.2 Wrapper around PHP's tokenizer ext... +phpunit/phpunit 3.7.37 The PHP Unit Testing framework. +phpunit/phpunit-mock-objects 1.2.3 Mock Object library for PHPUnit psr/log 1.0.0 Common interface for logging libra... ruflin/elastica 2.2.0 Elasticsearch Client +squizlabs/php_codesniffer 2.3.4 PHP_CodeSniffer tokenizes PHP, Jav... symfony/process v2.7.3 Symfony Process Component +symfony/yaml v2.8.0 Symfony Yaml Component wikimedia/assert v0.2.2 Provides runtime assertions wikimedia/avro v1.7.7 A library for using Apache Avro wi... wikimedia/base-convert v1.0.1 Convert an arbitrarily-long string...
Comment Actions
@Krinkle would it be possible to do what phpcs does and specify a script in the test.
Comment Actions
We already do that and that works fine for simple projects. We'll keep that as is. The problem here is that we need to combine secured dependencies from mediawiki/vendor.git with fresh development tools from Composer in Jenkins. The actual execution of the test (in composer.json) is simple and that problem is already solved.
Comment Actions
Change 270488 had a related patch set uploaded (by Krinkle):
Add bin/mw-fetch-composer-dev.sh script
Comment Actions
Change 270489 had a related patch set uploaded (by Krinkle):
[WIP] Add mw-fetch-composer-dev to MediaWiki phpunit jobs
This comment was removed by Krinkle.
Comment Actions
The preparation work that doesn't affect anything yet is finished.
The following three patches should:
- Make MediaWiki phpunit jobs always install require-dev packages from Composer.
- Use the local vendor PHPUnit install instead of the old integration install. (Note, this doesn't upgrade it yet, it just makes it use the local one, which we can then trivially change by updating the package version in composer.json)
- https://gerrit.wikimedia.org/r/270489 - [integration/config] Add mw-fetch-composer-dev to MediaWiki phpunit jobs
- https://gerrit.wikimedia.org/r/270515 - [integration/slave-scripts] mw-phpunit: Remove default value for PHPUNIT_DIR
- https://gerrit.wikimedia.org/r/270545 - [integration/config] mediawiki-core-code-coverage: Remove hardcoded phpunit-dir
Once that settles, we can close T99982: Upgrade PHPUnit to 4.0+ at any time by merging https://gerrit.wikimedia.org/r/270485 in MediaWiki core.
Comment Actions
Change 270515 had a related patch set uploaded (by Krinkle):
Remove default PHPUNIT_DIR mw-phpunit.sh and mw-run-phpunit-allexts.sh
Comment Actions
Change 270545 had a related patch set uploaded (by Krinkle):
mediawiki-core-code-coverage: Remove hardcoded phpunit-dir
Comment Actions
Change 270489 merged by jenkins-bot:
Add mw-fetch-composer-dev to MediaWiki phpunit jobs
Comment Actions
- https://gerrit.wikimedia.org/r/270489 - [integration/config] Add mw-fetch-composer-dev to MediaWiki phpunit jobs
This is now merged. The affected Jenkins jobs have been recompiled and deployed.
Contrary to what I thought this first patch already makes MediaWiki jobs use PHPUnit from Composer. Whilst with-phpunitdir is still hardcoded in the jobs, it doesn't end up being jobs now because PHPUnit first checks class_exists before utilising the include_dir paths. This check now returns true because the Composer autoloader happily provides PHPUnit. Great!
Verified so far:
- MediaWiki core patch (master branch)
- MediaWiki extension patch (TemplateData)
- MediaWIki core patch (REL1_23 branch - predates Composer) - we don't actually have phpunit jobs for this branch anymore. If we create them, they'd work fine.
- Wikidata/Wikibase patches - Wikidata jobs were already using Composer and are unaffected.
Comment Actions
Change 270515 merged by jenkins-bot:
Remove default PHPUNIT_DIR mw-phpunit.sh and mw-run-phpunit-allexts.sh
Comment Actions
Change 270545 merged by jenkins-bot:
mediawiki-core-code-coverage: Remove hardcoded phpunit-dir