Page MenuHomePhabricator

Support installing composer require-dev packages together with mediawiki/vendor
Closed, ResolvedPublic

Description

Handle composer require-dev together with mediawiki-vendor without overwriting the things that are already provided by mediawiki-vendor. Or find a different solution for how to have a phpunit around to run.

Related: T92605: Come up with non sucky solution for running "composer test" on repos that have vendor/ checked in

Event Timeline

JanZerebecki raised the priority of this task from to Medium.
JanZerebecki updated the task description. (Show Details)

One way would be to implement a composer update --only-dev.

I was thinking of: clone mediawiki/vendor, add use composer-merge-plugin to include mediawiki/core's composer.json, and then run composer install, which should fetch new dev deps, but not bump anything that is already installed from vendor.

If one updated something in core but not vendor that would still lead to an update instead of a failed job. Doing that while merging the composer.json from vendor might work, as long as that composer.json only ever contains exact version requirements. If we stopped using exact versions there, we could still generate one from the composer.lock in vendor.

JanZerebecki updated the task description. (Show Details)Oct 2 2015, 6:43 PM
JanZerebecki set Security to None.

I was thinking of: clone mediawiki/vendor, add use composer-merge-plugin to include mediawiki/core's composer.json, and then run composer install, which should fetch new dev deps, but not bump anything that is already installed from vendor.

I was wondering the if the same thing would work. We could even extend composer-merge-plugin to allow configuration of what sections are allowed to be merged and setup mediawiki/vendor to only allow require-dev rather than everything.

Otherwise I think we would want the composer.lock from mediawiki/vendor to be used with the composer.json from mediawiki/core and ... something ... that would let us only bring in the require-dev libs and dependencies.

One way would be to implement a composer update --only-dev.

My version of composer (which I think is the latest) doesn't have an --only-dev flag for composer update.

The only way I know to pick and choose updates is with the composer update vendor/package syntax.

Krinkle renamed this task from handle composer require-dev together with mediawiki-vendor to Support installing composer require-dev packages together with mediawiki/vendor.Dec 15 2015, 11:02 PM

This is blocking various things:

  • Updating MediaWiki core to use PHPUnit specified in composer.json.
    • Which in turn blocks out ability to improve PHPUnit performance (newer versions are much more optimised) as well as newer PHPunit features and various bug fixes in its mocking library.
  • Improving MediaWIki core unit test performance by using vfsStream (virtual file system for PHP) because we can't specify it as a require-dev dependency since it isn't being installed.

And various other things that make use of composer for tooling around MediaWiki core.

The only way I know to pick and choose updates is with the composer update vendor/package syntax.

This might actually be pretty easy to script for Jenkins jobs:

  1. Clone mediawiki/core
  2. Clone mediawiki/vendor
  3. Parse the "require-dev" bits out of core/composer.json
  4. Install those packages explictly

Here's a POC script that shows one way to do that:

#!/usr/bin/env bash
set -xe

git clone --depth 1 https://github.com/wikimedia/mediawiki.git
cd mediawiki
git clone --depth 1 https://github.com/wikimedia/mediawiki-vendor.git vendor
cd vendor
composer show --installed > from-vendor
jq -r '.["require-dev"]|to_entries|map([.key,.value])[]|join("=")' ../composer.json | xargs composer require
composer show --installed > after-require
diff -uw from-vendor after-require

The result of the diff check at the end for me locally:

--- from-vendor 2015-12-15 16:53:48.000000000 -0700
+++ after-require       2015-12-15 16:53:55.000000000 -0700
@@ -1,16 +1,28 @@
 composer/semver                   1.2.0    Semver library that offers utiliti...
 cssjanus/cssjanus                 v1.1.1   Convert CSS stylesheets between le...
 firebase/php-jwt                  v3.0.0   A simple library to encode and dec...
+jakub-onderka/php-parallel-lint   v0.9     This tool check syntax of PHP file...
+justinrainbow/json-schema         1.5.0    A library to validate a json schema.
 kzykhys/pygments                  v1.0.0   A Thin Wrapper for the Python Pygm...
 liuggio/statsd-php-client         v1.0.18  Statsd (Object Oriented) client li...
 mediawiki/at-ease                 v1.1.0   Safe replacement to @ for suppress...
+mediawiki/mediawiki-codesniffer   v0.4.0   MediaWiki CodeSniffer Standards
 monolog/monolog                   1.17.2   Sends your logs to files, sockets,...
 nmred/kafka-php                   v0.1.4   Kafka client for php
 oojs/oojs-ui                      v0.14.1  Provides library of common widgets...
 oyejorge/less.php                 v1.7.0.9 PHP port of the Javascript version...
+phpunit/php-code-coverage         1.2.18   Library that provides collection, ...
+phpunit/php-file-iterator         1.4.1    FilterIterator implementation that...
+phpunit/php-text-template         1.2.1    Simple template engine.
+phpunit/php-timer                 1.0.7    Utility class for timing
+phpunit/php-token-stream          1.2.2    Wrapper around PHP's tokenizer ext...
+phpunit/phpunit                   3.7.37   The PHP Unit Testing framework.
+phpunit/phpunit-mock-objects      1.2.3    Mock Object library for PHPUnit
 psr/log                           1.0.0    Common interface for logging libra...
 ruflin/elastica                   2.2.0    Elasticsearch Client
+squizlabs/php_codesniffer         2.3.4    PHP_CodeSniffer tokenizes PHP, Jav...
 symfony/process                   v2.7.3   Symfony Process Component
+symfony/yaml                      v2.8.0   Symfony Yaml Component
 wikimedia/assert                  v0.2.2   Provides runtime assertions
 wikimedia/avro                    v1.7.7   A library for using Apache Avro wi...
 wikimedia/base-convert            v1.0.1   Convert an arbitrarily-long string...
daniel added a subscriber: daniel.Feb 13 2016, 2:17 PM

Any progress on this? It would be really nice if we could move to PHPUnit 4.

Krinkle claimed this task.Feb 13 2016, 2:29 PM

@daniel Giving this a shot over the weekend.

@Krinkle would it be possible to do what phpcs does and specify a script in the test.

@Krinkle would it be possible to do what phpcs does and specify a script in the test.

We already do that and that works fine for simple projects. We'll keep that as is. The problem here is that we need to combine secured dependencies from mediawiki/vendor.git with fresh development tools from Composer in Jenkins. The actual execution of the test (in composer.json) is simple and that problem is already solved.

Change 270488 had a related patch set uploaded (by Krinkle):
Add bin/mw-fetch-composer-dev.sh script

https://gerrit.wikimedia.org/r/270488

Change 270489 had a related patch set uploaded (by Krinkle):
[WIP] Add mw-fetch-composer-dev to MediaWiki phpunit jobs

https://gerrit.wikimedia.org/r/270489

This comment was removed by Krinkle.

Change 270488 merged by jenkins-bot:
Add bin/mw-fetch-composer-dev.sh script

https://gerrit.wikimedia.org/r/270488

The preparation work that doesn't affect anything yet is finished.

The following three patches should:

  • Make MediaWiki phpunit jobs always install require-dev packages from Composer.
  • Use the local vendor PHPUnit install instead of the old integration install. (Note, this doesn't upgrade it yet, it just makes it use the local one, which we can then trivially change by updating the package version in composer.json)
  1. https://gerrit.wikimedia.org/r/270489 - [integration/config] Add mw-fetch-composer-dev to MediaWiki phpunit jobs
  2. https://gerrit.wikimedia.org/r/270515 - [integration/slave-scripts] mw-phpunit: Remove default value for PHPUNIT_DIR
  3. https://gerrit.wikimedia.org/r/270545 - [integration/config] mediawiki-core-code-coverage: Remove hardcoded phpunit-dir

Once that settles, we can close T99982: Upgrade PHPUnit to 4.0+ at any time by merging https://gerrit.wikimedia.org/r/270485 in MediaWiki core.

Change 270515 had a related patch set uploaded (by Krinkle):
Remove default PHPUNIT_DIR mw-phpunit.sh and mw-run-phpunit-allexts.sh

https://gerrit.wikimedia.org/r/270515

Change 270545 had a related patch set uploaded (by Krinkle):
mediawiki-core-code-coverage: Remove hardcoded phpunit-dir

https://gerrit.wikimedia.org/r/270545

Change 270489 merged by jenkins-bot:
Add mw-fetch-composer-dev to MediaWiki phpunit jobs

https://gerrit.wikimedia.org/r/270489

Krinkle closed this task as Resolved.Feb 15 2016, 11:52 PM
  1. https://gerrit.wikimedia.org/r/270489 - [integration/config] Add mw-fetch-composer-dev to MediaWiki phpunit jobs

This is now merged. The affected Jenkins jobs have been recompiled and deployed.

Contrary to what I thought this first patch already makes MediaWiki jobs use PHPUnit from Composer. Whilst with-phpunitdir is still hardcoded in the jobs, it doesn't end up being jobs now because PHPUnit first checks class_exists before utilising the include_dir paths. This check now returns true because the Composer autoloader happily provides PHPUnit. Great!

Verified so far:

  • MediaWiki core patch (master branch)
  • MediaWiki extension patch (TemplateData)
  • MediaWIki core patch (REL1_23 branch - predates Composer) - we don't actually have phpunit jobs for this branch anymore. If we create them, they'd work fine.
  • Wikidata/Wikibase patches - Wikidata jobs were already using Composer and are unaffected.

Change 270515 merged by jenkins-bot:
Remove default PHPUNIT_DIR mw-phpunit.sh and mw-run-phpunit-allexts.sh

https://gerrit.wikimedia.org/r/270515

Change 270545 merged by jenkins-bot:
mediawiki-core-code-coverage: Remove hardcoded phpunit-dir

https://gerrit.wikimedia.org/r/270545