composer.json should be useable by WMF, core and extensions
Closed, ResolvedPublic

Description

We should look at way to generate or update composer.json. We may want to look at a way to manage composer.json in update.php, etc.


Version: 1.24rc
Severity: enhancement
See Also:
https://github.com/composer/composer/issues/2761

Details

Reference
bz65188
bzimport raised the priority of this task from to Normal.
bzimport set Reference to bz65188.
bzimport added a subscriber: Unknown Object (MLST).

Any changes that are made need to be usable by WMF and extension developers.

Brian Davis, Ori and Jeroen have already spent time on this. After a discussion today, we hope to finally produce a use of compooser.php in MediaWiki that all will be able to use and appreciate.

mwalker wrote:

Per Jeroen, we need to have the ability for end users (using the tarball or otherwise) to composer require $package without having to manually run through a horrible merge on update of mediawiki core.

However, core requires composer because we're now starting to pull in external dependencies (like MonoLog / PSR3 logging). This requires that core have a composer.json file!

We've resolved that the installer and the update scripts need to be aware of composer.json files and be able to merge new with old.

We've also resolved that it is undesirable to ship a composer.json file with the tarball. Though; it was not agreed on if having it in git was acceptable. (See https://gerrit.wikimedia.org/r/#/c/132788/).

It seems that the path forward is to write a hopefully simple stub around composer in the install and update scripts. This stub would merge the core dependencies file (I propose composer.core.json) into a non versioned composer.json. Then, the scripts would either run composer update or leave that up to the end user.

We've resolved that so long as mediawiki does a best effort to merge the files that it is up to the end user to resolve any version dependency problems.

ori added a comment.May 18 2014, 6:03 PM

There's a way to achieve this with composer:
https://getcomposer.org/doc/articles/custom-installers.md

There's a very good example here:
https://github.com/phpDocumentor/UnifiedAssetInstaller

My rough sense of how this would work is this:

MediaWiki's composer.json would specify a custom type and declare a dependency on the composer/installers package:

{
    "name": "mediawiki/core",
    "type": "mediawiki-instance",
    "require": {
        "composer/installers": "~1.0"
    }
}

We'd implement a mediawiki-instance installer (like https://github.com/composer/installers/blob/master/src/Composer/Installers/MediaWikiInstaller.php is a mediawiki-plugin installer).

The installer would merge additional requirements from an extensions.json file, if present.

A skeleton extensions.json would be checked into git but then added to gitignore so that subsequent modifications don't dirty the repo state.

If custom installers are that flexible, sounds like it's flexible enough that it could merge in the library dependencies of extensions even if they weren't directly installed using composer.

Unknown Object (User) added a comment.May 18 2014, 6:29 PM

I'm not really sure what you are trying to do with a extensions.json but please keep in mind that is not only extensions with type "mediawiki-extension" we are talking about. We also support type "mediawiki-skin" [0] that allows to install skin related extension via Composer.

The installer would merge additional requirements from an extensions.json file, if present.

Just to be clear, an extensions.json is something MW does on its own right?

An extension should only deploy a composer.json (because extensions.json is a not supported json and is only relevant to an internal MW process) as this is the official support declaration of dependencies via Composer.

[0] https://packagist.org/packages/mediawiki/chameleon-skin

(In reply to Ori Livneh from comment #3)

A skeleton extensions.json would be checked into git but then added to
gitignore so that subsequent modifications don't dirty the repo state.

We'd have to call it extensions-example.json, because once a file is added to the repository, gitignore will no longer affect it.

(In reply to Ori Livneh from comment #3)

We'd implement a mediawiki-instance installer (like
<https://github.com/composer/installers/blob/master/src/Composer/Installers/
MediaWikiInstaller.php> is a mediawiki-plugin installer).

The installer would merge additional requirements from an extensions.json
file, if present.

I think this is a pretty viable solution. My only concern is that now people would have to manually edit extensions.json in order to install new extensions.

Which is why maybe we should take the Vagrant approach! It seems the Composer\Plugin\PluginInterface is meant to install plugins into Composer. Do we know if it would be possible to add custom commands to composer, similar to how we add "list-roles" with Vagrant?

ori added a comment.May 18 2014, 8:01 PM

(In reply to Tyler Romeo from comment #6)

Do we know if it would be possible to add custom commands to composer,
similar to how we add "list-roles" with Vagrant?

It's in the works: https://github.com/composer/composer/issues/2761

(In reply to Ori Livneh from comment #7)

(In reply to Tyler Romeo from comment #6)
> Do we know if it would be possible to add custom commands to composer,
> similar to how we add "list-roles" with Vagrant?

It's in the works: https://github.com/composer/composer/issues/2761

This issue now has a pull request proposed: https://github.com/composer/composer/pull/3377

There has been discussion on https://gerrit.wikimedia.org/r/#/c/173429/ related to this topic. The conversation there is centered around the challenges of tracking composer dependencies for extensions that have not been installed via composer (in this case via git clone) but the problems are closely related to those of a user wanting to manage local composer dependencies and track MediaWiki upstream dependencies without conflicts.

@Legoktm said:

We could have a maintenance script that goes through a provided list of extension directories, reads the dependencies from the composer.json files if they exist, and creates a vendor/composer.json with all of them (the equivalent of mediawiki/vendor's composer.json) and runs composer update. Would that be helpful and make this easier @Nikerabbit?

@Legoktm said:

*And include core's composer.json too. Basically it would create a "mediawiki/vendor" for your wiki.

@Nikerabbit said:

As far as I know you cannot tell composer which file to use as composer.json, so unless you can do that with some composer hook trickery, you will have the problem of having merge conflicts for the composer.json file when updating core.

@bd808 said:

As far as I know you cannot tell composer which file to use as composer.json

@Legoktm's suggestion of using $IP/vendor/composer.json would work for this. You can tell composer in a composer.json file where to put the vendored files. In this case you would tell it to vendor to '.' which would make $IP/vendor look like MW-core wants it to. This scheme would require the user to never run composer in $IP, but otherwise should work. We could probably even put some sort of hook into the MW-core composer.json that would warn/abort if $IP/vendor/composer.json was found.

Tagging with the Librarization and #1.25.0_release projects not as cookie licking but to ensure that this issue stays visible to those efforts.

Kghbln added a subscriber: Kghbln.Dec 3 2014, 10:54 PM

I'm going to take a shot at a solution similar to the one proposed by Ori in T67188#700991 but using a plugin rather than a complete installer.

My idea is to create a plugin that allows the composer.json shipped with mediawiki/core to specify a list of additional composer.json formatted files to be merged into the Composer configuration at runtime. Using this functionality we can add configuration to our composer.json that looks for one or more additional configuration files that are managed by the local deployer. We can also pull in the composer.json files from any extensions and skins that have been installed as git clones so that their dependencies are installed in the top level $IP/vendor directory with a single composer update invocation.

I have a working version if this plugin and hope to publish it to packagist soon for wider testing. Assuming that works we can add it to mediawiki/core's composer.json and iterate from there to resolve this issue.

bd808 moved this task from Untriaged to In Dev on the Librarization board.
bd808 moved this task from Backlog to In Dev/Progress on the MediaWiki-Core-Team board.

@bd808, thanks for working on this! If you need testers or someone to try this with some extensions, let me know.

I would also be interested to test this at translatewiki.net - in fact I already tried it but it doesn't seem to work yet.

[ErrorException] Invalid argument supplied for foreach()  

Exception trace:
 () at /resources/nike/ctest/mediawiki/vendor/wikimedia/composer-merge-plugin/src/MergePlugin.php:215
 Composer\Util\ErrorHandler::handle() at /resources/nike/ctest/mediawiki/vendor/wikimedia/composer-merge-plugin/src/MergePlugin.php:215
 Wikimedia\Composer\MergePlugin->onDependencySolve() at n/a:n/a

Extensions installed via composer are also installed to extensions/* and have their composer files at extensions/*/composer.json. Loading those seems unnecessary, but maybe not that harmful?

fbstj added a subscriber: fbstj.Dec 23 2014, 4:59 PM

The plugin is on packagist now (https://packagist.org/packages/wikimedia/composer-merge-plugin) so it should be a bit easier to test. I found and fixed a couple of copy-n-paste bugs that may have caused it to fail for @Nikerabbit.

I've worked out quite a few kinks in the merge plugin. I have it working well now in my local dev environment and would welcome additional testers.

First I installed the plugin:

$ composer require wikimedia/composer-merge-plugin dev-master

Then I configured my $IP/composer.json to merge a local settings file as well as the composer.json files from any extensions that I have cloned:

"extra": {
    "merge-plugin": {
        "include": [
            "composer.local.json",
            "extensions/*/composer.json"
         ]
    }
}

My composer.local.json is:

{
    "require": {
        "monolog/monolog": "1.11.0"
    }
}

Finally I ran composer -v update:

$ composer -v update
  [merge] Loading composer.local.json...
  [merge] Merging monolog/monolog
  [merge] Loading extensions/AntiSpoof/composer.json...
  [merge] Merging composer/installers
  [merge] Loading extensions/Elastica/composer.json...
  [merge] Merging ruflin/elastica
  [merge] Loading extensions/Scribunto/composer.json...
  [merge] Deferring duplicate composer/installers
  [merge] Loading extensions/Translate/composer.json...
  [merge] Deferring duplicate php
  [merge] Deferring duplicate composer/installers
  [merge] Merging mediawiki/universal-language-selector
  [merge] Loading extensions/UniversalLanguageSelector/composer.json...
  [merge] Deferring duplicate php
  [merge] Deferring duplicate composer/installers
  [merge] Loading extensions/WikiEditor/composer.json...
  [merge] Deferring duplicate composer/installers
Loading composer repositories with package information
Updating dependencies (including require-dev)
  [merge] Adding dependency mediawiki/scribunto requires composer/installers (>= 1.0.1.0)
  [merge] Adding dependency mediawiki/translate requires php (>= 5.3.0.0)
  [merge] Adding dependency mediawiki/translate requires composer/installers (>= 1.0.1.0)
  [merge] Adding dependency mediawiki/universal-language-selector requires php (
>= 5.3.0.0)
  [merge] Adding dependency mediawiki/universal-language-selector requires compo
ser/installers (>= 1.0.1.0)
  [merge] Adding dependency mediawiki/wiki-editor requires composer/installers (
[])
  - Installing composer/installers (v1.0.19)
    Loading from cache
    Extracting archive

  - Installing monolog/monolog (1.11.0)
    Loading from cache
    Extracting archive

  - Installing ruflin/elastica (v1.3.0.0)
    Loading from cache
    Extracting archive

  - Installing mediawiki/universal-language-selector (2014.11)
    Loading from cache
    Extracting archive

monolog/monolog suggests installing graylog2/gelf-php (Allow sending log messages to a GrayLog2 server)
monolog/monolog suggests installing raven/raven (Allow sending log messages to a Sentry server)
monolog/monolog suggests installing doctrine/couchdb (Allow sending log messages to a CouchDB server)
monolog/monolog suggests installing videlalvaro/php-amqplib (Allow sending log messages to an AMQP server using php-amqplib)
monolog/monolog suggests installing ext-amqp (Allow sending log messages to an AMQP server (1.0+ required))
monolog/monolog suggests installing ext-mongo (Allow sending log messages to a MongoDB server)
monolog/monolog suggests installing aws/aws-sdk-php (Allow sending log messages to AWS services like DynamoDB)
monolog/monolog suggests installing rollbar/rollbar (Allow sending log messages to Rollbar)
ruflin/elastica suggests installing munkie/elasticsearch-thrift-php (Allow using thrift transport)
ruflin/elastica suggests installing guzzlehttp/guzzle (Allow using guzzle 4.x as the http transport (requires php 5.4))
mediawiki/universal-language-selector suggests installing mediawiki/cldr (Language names in all languages)
  [merge] Adding dependency mediawiki/scribunto requires composer/installers (>= 1.0.1.0)
  [merge] Adding dependency mediawiki/translate requires php (>= 5.3.0.0)
  [merge] Adding dependency mediawiki/translate requires composer/installers (>= 1.0.1.0)
  [merge] Adding dependency mediawiki/universal-language-selector requires php (>= 5.3.0.0)
  [merge] Adding dependency mediawiki/universal-language-selector requires composer/installers (>= 1.0.1.0)
  [merge] Adding dependency mediawiki/wiki-editor requires composer/installers ([])
Writing lock file
Generating optimized autoload files
bd808 moved this task from In Dev to Needs Review on the Librarization board.Dec 29 2014, 6:12 PM

Change 182323 had a related patch set uploaded (by BryanDavis):
Introduce Composer merge plugin

https://gerrit.wikimedia.org/r/182323

Patch-For-Review

Change 182323 merged by jenkins-bot:
Introduce Composer merge plugin

https://gerrit.wikimedia.org/r/182323

bd808 moved this task from Needs Review to Done on the Librarization board.Jan 8 2015, 11:36 PM

Does the addition of my merge plugin resolve this issue entirely or are there additional steps that need to be taken? composer require $package will still make a dirty diff with future changes that come from upstream in MediaWiki, but a user can manually move the require from composer.json to composer.local.json to alleviate this.

I suppose there should at the very least be some documentation added somewhere about the capability.

As a matter of tracking, here is the pull request to allow plugins to add custom commands to composer:

https://github.com/composer/composer/issues/1234

Legoktm closed this task as Resolved.Jan 28 2015, 9:49 PM

Does the addition of my merge plugin resolve this issue entirely or are there additional steps that need to be taken? composer require $package will still make a dirty diff with future changes that come from upstream in MediaWiki, but a user can manually move the require from composer.json to composer.local.json to alleviate this.

I think so. composer.json itself is not usable by others, but there is an alternative avenue for adding dependencies through composer.local.json

I suppose there should at the very least be some documentation added somewhere about the capability.

https://www.mediawiki.org/w/index.php?title=Composer%2FFor_extensions&diff=1380007&oldid=1357390

bd808 moved this task from Done to Archive on the MediaWiki-Core-Team board.Feb 5 2015, 1:58 AM

Not sure this is fully resolved given my concerns in https://lists.wikimedia.org/pipermail/wikitech-l/2015-July/082564.html and https://lists.wikimedia.org/pipermail/wikitech-l/2015-July/082568.html. As is, this is likely to cause problems for third party users that are using extensions that have composer dependencies that overlap with those of MediaWiki.