Users should be able to customize the root composer.json file however they would like.

This premise lacks support.

At the same time, the composer.lock file is not being tracked in the project, this has forced the pinning of versions in composer.json.

According to https://www.mediawiki.org/wiki/Manual:External_libraries, the reasoning for pinning is not "because composer.lock is not being tracked". It's "we don't trust upstream to follow semver".

There also seems to be multiple /vendor folders within a MediaWiki project (i.e. within an extension), which could lead to multiple copies (and perhaps incompatible versions) within a single project. Ideally there should only be a single /vendor directory.

It's possible to have a single /vendor if you use composer-merge-plugin to import the extensions' composer.json from your composer.local.json. I do this for my local development environment.

The main annoyance I've found is that AbuseFilter declares a composer dependency on AntiSpoof, which makes composer insist on overwriting my local extensions/AntiSpoof git checkout with its own. Some other extensions have the same issue.

Also, the root composer.json and composer.lock can be tracked since the user will have their own git repo and /core will be updated with composer.

If the user is going to have their own (forked) git repo, couldn't they modify the root composer.json as things are already?

In T166956#3332480, @Anomie wrote:

Users should be able to customize the root composer.json file however they would like.

This premise lacks support.

It really depends on what the "root" of MediaWiki is. Is this closer to a user's "project" (that can be modified) or closer to a "library" (that ought not ever be modified)?
See: https://getcomposer.org/doc/04-schema.md#type

As examples:

Drupal is a project: https://github.com/drupal/drupal/blob/8.4.x/composer.json
with a core library: https://github.com/drupal/drupal/blob/8.4.x/core/composer.json

Symfony Standard is a project: https://github.com/symfony/symfony-standard/blob/3.3/composer.json
with a core library: https://github.com/symfony/symfony/blob/master/composer.json

Laravel is a project: https://github.com/laravel/laravel/blob/master/composer.json
with a core library: https://github.com/laravel/framework/blob/5.4/composer.json

According to https://github.com/wikimedia/mediawiki/blob/master/composer.json
MediaWiki has no type. I think the proper type is project, however this creates a problem because then the composer.json (and anything in the root for that matter), is by definition, owned by the user rather than the library.

At the same time, the composer.lock file is not being tracked in the project, this has forced the pinning of versions in composer.json.

According to https://www.mediawiki.org/wiki/Manual:External_libraries, the reasoning for pinning is not "because composer.lock is not being tracked". It's "we don't trust upstream to follow semver".

Oh I completely agree with that, but it should be pinned in composer.lock (already is) not composer.json. All developer (and build tools) should run composer install (to use compser.lock exclusively) rather than composer update. If we want to support multiple versions of a library, we should allow users to install different versions that we MediaWiki should work with, but we're only guaranteeing it works with what's in the lock file.

For instance, if library X comes out with a security update, a user should be able to execute composer updateand get the patch immediately. Right now, they would have to wait for MediaWiki to update the composer.json file (or manually update it themselves).

There also seems to be multiple /vendor folders within a MediaWiki project (i.e. within an extension), which could lead to multiple copies (and perhaps incompatible versions) within a single project. Ideally there should only be a single /vendor directory.

It's possible to have a single /vendor if you use composer-merge-plugin to import the extensions' composer.json from your composer.local.json. I do this for my local development environment.

The main annoyance I've found is that AbuseFilter declares a composer dependency on AntiSpoof, which makes composer insist on overwriting my local extensions/AntiSpoof git checkout with its own. Some other extensions have the same issue.

Good to know about the work around, I didn't even think about that. Though I feel like it would be more straightforward to just declare them as dependencies in the root and use the --prefer-source option if you need it cloned.

Also, the root composer.json and composer.lock can be tracked since the user will have their own git repo and /core will be updated with composer.

If the user is going to have their own (forked) git repo, couldn't they modify the root composer.json as things are already?

They certainly could, but then they'd have to merge any changes when updating. :(

Changing what the MediaWiki repository "is" and how it behaves is quite scary.

If the main motivation is the ability to update MediaWiki via composer, I suggest to just create a new repository with the appropriate composer.json. Call it MediaWiki-Composer-Wrapper or something.

As to installing extensions via composer: that was declined based on the fact that the relationship between an application and its plugins is different from the relationship between an application and the libraries it needs. Treating a web application as a library needed by a plugin is awkward and misleading. So a different mechanism would be required for managing extensions. Could be a composer plugin, perhaps.

I support the idea of having a clear separation between locally editable vs versioned/deployed files. But I think we have to be extremely careful not to break 3rd party tooling. If anything, I recommend to move the versioned files, and leave the editable files where they are.

In T166956#3380018, @daniel wrote:

Changing what the MediaWiki repository "is" and how it behaves is quite scary.

We would only be changing it in regards to Composer. Everything else would behave exactly the same as before.

If the main motivation is the ability to update MediaWiki via composer, I suggest to just create a new repository with the appropriate composer.json. Call it MediaWiki-Composer-Wrapper or something.

There are a lot of reasons to do this that have nothing to do with Composer that I've outlined in the issue summary. One of those is Composer. While a wrapper might work, it doesn't resolve the other issues, and it creates some others (like having extensions in more than one directory, etc.). It also creates another thing that has to be maintained.

As to installing extensions via composer: that was declined based on the fact that the relationship between an application and its plugins is different from the relationship between an application and the libraries it needs. Treating a web application as a library needed by a plugin is awkward and misleading. So a different mechanism would be required for managing extensions. Could be a composer plugin, perhaps.

I don't think it's a library at all, it is an application. But Drupal is also an application and the application is loaded with Composer from your project. They don't need a different mechanism to manage extensions, why would we?

I support the idea of having a clear separation between locally editable vs versioned/deployed files. But I think we have to be extremely careful not to break 3rd party tooling. If anything, I recommend to move the versioned files, and leave the editable files where they are.

I've outlined in T167038 exactly that!

For reference, see the declined RFC T467: RfC: Extension management with Composer

For discussion in next week's TechCom internal meeting

This task no longer requires any changes to the MediaWiki code base and would be creating a new "wrapper" project. Therefor removing TechCom-RFC.

@dbarratt Creating such a wrapper project indeed does not need an RFC. Proposing the structure defined by that wrapper for new 3rd party installations, or for wmf deployments, or for the standard development environment - that would need an RFC.

In T166956#4070369, @daniel wrote:

@dbarratt Creating such a wrapper project indeed does not need an RFC. Proposing the structure defined by that wrapper for new 3rd party installations, or for wmf deployments, or for the standard development environment - that would need an RFC.

That's fine, it needs to be created first, and later someone else can identify whether it should be recommended or used within wmf.

@dbarratt Thanks, trying this out in a personal repository (e.g. on GitHub) is fine indeed.

We only ask that, if it is published to Packagist, that it not be under the wikimedia name. Instead it should use either mediawiki/ , or a personal vendor name. If using mediawiki/, please use a more descriptive name specific to this purpose, and not mediawiki/mediawiki, mediawiki/core, or mediawiki/mediawiki-project. That is until/unless it becomes official.

For example dbarrat/mediawiki or mediawiki/composer-mediawiki-project.

I don't know why we wouldn't put the repo on the wikimedia org on GitHub (or on Gerrit, I have no preference) and allow it to be managed by WMF (including the item on packagist). In this way, it's fine to use the mediawiki/mediawiki namespace since it can be repossessed for another purpose later if we determine this is of no value, but if it works out well, there wont be a pointless breaking change.

I plan on doing this at Wikimedia-Hackathon-2018, I don't think think it will take very long.

@dbarratt To clarify, I was only referring to package names for Packagist. (Not code hosting, or PHP namespaces, just in case). It's fine on Gerrit indeed, and would make sense. However, during the experiment phase it should not be published to Packagist under a generic name. As mentioned, a more specific package name on Packagist is fine under mediawiki/.

In T166956#4071684, @Krinkle wrote:

@dbarratt To clarify, I was only referring to package names for Packagist. (Not code hosting, or PHP namespaces, just in case). It's fine on Gerrit indeed, and would make sense. However, during the experiment phase it should not be published to Packagist under a generic name. As mentioned, a more specific package name on Packagist is fine under mediawiki/.

I still do not understand what the problem is with using mediawiki/mediawiki (which can be deleted or repossessed at anytime), we already have a docker image that we do not "officially" support:
https://hub.docker.com/_/mediawiki/

so wrapping MediaWiki in an alternative (not officially supported) method, is not a big deal in my mind. By saying it's unofficial, we're just saying we don't provide support for it, nor are we commuting to it existing in the future.

Just my personal opinion, but I am also strongly opposed to using mediawiki/mediawiki, wikimedia/mediawiki etc as that implies its official, and we'll start getting support questions about it, etc.

Perhaps mediawiki/project ? then the command would be:

composer create-project mediawiki/project

which I think makes it somewhat more clear that it's a starting point. I mean mediawiki/starter would also work. I don't have a huge preference. Another option (similar to Symfony) would be mediawiki/skeleton but I don't like that one as well.

Release Engineering is interested in this work for the purposes of CI and the new deployment pipeline. The current approach, which is to rely on test dependencies defined via a Python global in integration/config, is certainly not ideal and not viable for the pipeline. See T193824: Determine a standard way of installing MediaWiki lib/extension dependencies within containers for possible solutions we've been discussing.

You can now test this out by executing:

composer create-project davidbarratt/mediawiki

For now, the code is located at:
https://github.com/davidbarratt/mediawiki

This will be moved to gerrit when the request is approved, then it will be added to Packagist under the mediawiki namspace... perhaps mediawiki/project ?

Change 434764 had a related patch set uploaded (by Dbarratt; owner: Dbarratt):
[mediawiki/composer/project@master] Merge changes from davidbarratt/mediawiki.

https://gerrit.wikimedia.org/r/434764

Task description

It is difficult to manage a MediaWiki project with composer.

This is a specific solution (managing a project with composer), not a use case or problem. What kinds of actions would you like to be able to perform when managing a MediaWiki project? But in its current form the problem sounds like "I want to use X and the problem is I can't use X", which is difficult to host an RFC discussion around.

I think it would be fine to be a bit repetitive and basically mention most of what Composer CLI does. I'm also not saying we won't use it. I'd just like the outcome to be more concrete so that we can weigh one implementation against another. And if someone proposes to make something custom, then part of the cost we'd weigh is that it has to be developed and maintained, and that it'd be inconsistent with what people may've learned elsewhere, etc.

Task description

While you can edit the composer.local.json you cannot use the Composer CLI.

Are there abilities available to composer.json not available to composer.local.json? I'm not proposing that that would suffice if the case, just want to get all the facts available :)

@Krinkle I updated the task description, is that helpful?

OK. So the use case is for site administrators for setting up a new MediaWiki-based web site.

Some differences:

• To get started, run the CLI command composer create-project …. Currently the command git clone ….
• To upgrade core or an extension, edit composer.json through a Composer command that e.g. bumps or re-evaluates the required version of mediawiki/core. Currently the command git checkout (to switch tags) or git pull (to forward a branch).
• …

Some things that are the same:

• Download extensions that don't have composer deps with one command. Either composer require or git clone.
• Install extensions by adding wfLoadExtension calls to LocalSettings.php as-needed.
• Install the database schema and create initial settings file by running maintenance/install.php (or web UI).
• Update the database schema by running maintenance/update.php (or web UI).
• …

Some benefits:

• Run one command to get started with plain MediaWiki core (composer create-project …) instead of two (git clone 2x, or git clone and composer install).
• For extensions that have composer dependencies, download and install their deps with one command (composer require …), instead of two commands and editing a file (git clone, edit composer.local.json`, composer install).

Some downsides:

• Long-term cost of maintaining, testing and documenting mediawiki/core on Composer. (Could be isolated to a single team.)
• Long-term cost of maintaining, testing and documenting mediawiki-website on Composer. (Could be isolated to a single team.)
• Offering yet another way to install MediaWiki. This increases choice paralysis for site admins.
• Supporting yet another way to operate MediaWiki. This is also a cost that cannot be isolated given it can affect developers/products everywhere when users report issues with it. It could be delegated once the common aspect is understood, but this is not automatic or free.

Questions:

• Is this method of operating a MW site aimed mainly at stable installs, or is it something you'd also consider useful as development environments? My experience with git repos and composer is that it hasn't been very practical to mix these two, but I may've done it incorrectly and/or things may've changed since.

@dbarratt Could you reflect on some of these points? Check if I understood things correctly, and/or offer additional points. Thanks!

In T166956#5059959, @Krinkle wrote:

• To upgrade core or an extension, edit composer.json through a Composer command that e.g. bumps or re-evaluates the required version of mediawiki/core. Currently the command git checkout (to switch tags) or git pull (to forward a branch).

The bennefit here would be that you don't get in a scenario where you have mismatching dependencies between core, your extensions/skins, or any of their dependencies if you use Composer. If you use git that could happen (though the update script mostly forces you to deal with this by updating everything).

• Install the database schema and create initial settings file by running maintenance/install.php (or web UI).
• Update the database schema by running maintenance/update.php (or web UI).

With the current prototype I created, there is a problem with the maintenance scripts (see: T195082) and the installer (see: T194880). However, in general, yes, they should work the same way.

Some benefits:

As well as using a system to keep all of the dependencies in versions that are compatible with each other. For instance, if you have two extensions A and B. If A requires guzzlehttp/guzzle in ^6.0.0 and B requires guzzlehttp/guzzle in ~6.2.0 Composer will install 6.2.3 instead of the latest version which is 6.3.3. It does this because B has not declared that it is compatible with anything other than 6.2.x, but A is fine with anything in 6.x.x. If it can't resolve the version to something that is compatible with everything Composer will throw an exception and force you to deal with the problem. This is a huge safe-guard against incompatible versions that we are not using, and we are forcing site administrators to deal with on their own.

• Long-term cost of maintaining, testing and documenting mediawiki/core on Composer. (Could be isolated to a single team.)
• Long-term cost of maintaining, testing and documenting mediawiki-website on Composer. (Could be isolated to a single team.)

I don't know if this is a huge concern. Honestly, I would recommend bundling it all together the way Drupal does, see drupal/core and drupal/drupal. This allows them to (1) maintain it as a single package and (2) use composer in the CI and packaging systems.

This is what I would recommend doing, but it involves moving everything in core down a level. The alternative is to make two separate packages (which is what I did in the prototype). The former is easier to maintain, but the latter is easier to implement.

• Offering yet another way to install MediaWiki. This increases choice paralysis for site admins.

I mean it's only a valid way to install it if you know how to use the command line (until the day someone comes up with a GUI for Composer). If you do, I would recommend making it the recommended way to install MediaWiki since the benefits of using Composer over git are pretty significant.

• Supporting yet another way to operate MediaWiki. This is also a cost that cannot be isolated given it can affect developers/products everywhere when users report issues with it. It could be delegated once the common aspect is understood, but this is not automatic or free.

See my comments above. I think it should be the preferred way to install and operate MediaWiki as it provides a safe-guard that other methods do not provide.

• Is this method of operating a MW site aimed mainly at stable installs, or is it something you'd also consider useful as development environments? My experience with git repos and composer is that it hasn't been very practical to mix these two, but I may've done it incorrectly and/or things may've changed since.

Personally, I prefer using Composer for development. You can use --prefer-source option and it will do a git clone rather than a tar download of the package. It will also warn you that the repo is "dirty" before you do a composer update or anything like that that would wipe out your changes. It's really useful if you are dealing with a project that has a lot of extensions or dependencies and you might need to work on any one of them. It also makes it way easier to work on libraries as you can use that option and then go into the folder and do your work and submit the patch without having to copy a bunch of things back and forth.

So I use it for both site admin (as a way to not have all the dependencies in my repos, as well as a versioning safe-guard) as well as for development.