Page MenuHomePhabricator

RFC: Hybrid extension management
Open, MediumPublic

Assigned To
None
Authored By
MarkAHershberger
Apr 16 2020, 4:41 PM
Referenced Files
None
Tokens
"Like" token, awarded by Holybiber."Like" token, awarded by Rudloff."Like" token, awarded by Akuckartz."Like" token, awarded by MarkAHershberger."Like" token, awarded by CCicalese_WMF."Love" token, awarded by dbarratt."Like" token, awarded by RHeigl."Like" token, awarded by freephile."Like" token, awarded by Kghbln."Like" token, awarded by JeroenDeDauw.

Description

Motivation

Over the last years there have been several attempts to use the Composer "package manager" as a way to manage MediaWiki extensions. At the moment there are already a lot of extensions that support Composer as a way of installing and updating.

The main concerns about using Composer as an extension-management-tool is that Composer "is for managing libraries, not extensions" and that extensions would bypass wfLoadExtension(). This second part was a special concern for multiple wikis hosted from the same codebase (i.e. wikifarms).

Requirements

The responsibility of "managing" extensions should be split up between Composer and MediaWiki's ExtensionRegistry. It would become a "hybrid system".

  • Composer can be used to download and update extensions and their libraries, but does not enable the extension
  • ExtensionRegistry is responsible to check for inter-extension-dependencies as well as for version constraints (e.g. to the core MediaWiki version)

This means that an incompatible or unsatisfied extension may be downloaded by Composer but ExtensionRegistry would not install it.

  • composer.json => Evaluated when downloading or updating source code.
  • extension.json => Evaluated when running MW (installer/updater, and page views).
Advantages
  • Acceptance by the community: Composer is already widely used. Other open source communities also use it for "module-management", not just library-management
  • Centralized vendor/ directory: Extensions can store their dependencies in a centralized directory, rather than having them in their respective sub-directory. Therefore multiple different versions of library code can be prevented. Also, Special:ExtensionDistributior seems not to handle libraries at all at the moment (T215713)
  • Support for "SemVer" and "release branch compatibility model": Composer can use SemVer versions as well as release branches ("dev-REL1_*"). This makes it easy for a user to setup a reliable update mechanism (e.g. by "subscribing" to the "REL1_*" branches only).

Exploration

The current situation can easily be improved by some minor policy and software changes:

Changes on the software
  • ExtensionRegistry would throw an exception if requirements specified in extension.json were not met. Separate exception handling code (like SMW's SetupCheck) would be used for exceptions that happen during these early checks to ensure that administrators get better information than the WSOD.
Changes on the policy
  • Extensions must not enable themselves automatically via installation by Composer (e.g. by autoload.files)
  • Extensions must only declare dependencies to libraries in the composer.json. All dependencies to MediaWiki core versions and inter-extension-dependencies must be declared in the extension.json (By this, a virtual MediaWiki package is not required).
  • The WMF should not just allow, but encourage the use of "packagist-compatible" (must have fields like name, type and extra.installer-name) composer.json files in the extension repositories.
Potential action items
  • Add composer-guidelines to the official MediaWiki extension developer resources (draft).
  • Improve ExtensionRegistry dependency error handling (to be discussed)

See also

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I strongly support this RFC. It will serve not only to document best practices for the use of composer for MediaWiki extensions and skins, but will also replace T467 with a policy that better reflects the current recommended state of the practice. I'll repeat my comment from T249573 below for the record.

The situation with respect to use of composer in extensions has changed significantly since T467. I used to be opposed to composer use in extensions myself, but I have been convinced by seeing how composer is currently used in extensions. Today, the use of composer by some third parties to install extensions and maintain their infrastructure is quite impressive.

One of the early objections to composer use had to do with bypassing extension registration and using composer to enable extensions, but even Semantic MediaWiki has changed so that it no longer automatically enables itself. It is important to differentiate between installing an extension, which can supported in many ways, and enabling an extension.

There are many third party MediaWiki extension developers who are using composer in ways that greatly enhance their productivity and the quality of their code. There are things that composer should definitely NOT be used for (e.g. enabling extensions, expressing extension dependencies on other extensions, or in most cases autoloading code), and that current extensions should be updated in light of, but there are aspects of composer that are quite beneficial.

Since there has not been any similar functionality added to MediaWiki itself (nor should there be if an existing open source solution exists to provide that functionality), it seems reasonable that, rather than banning the use of composer, there be an accepted, well-documented approach to how extensions can use composer. Composer support should be optional, and it is fine if there is a limitation on extensions loaded in the Wikimedia infrastructure that prevents extension installation with composer.

Third party developers that I have spoken to are willing to make changes in their use of composer to comply with a reasonable policy when accepted, but would be dramatically against any attempt to remove all support for composer to install extensions.

We could make the next Wednesday's TechCom office hour about this RFC. Traditionally, the first meeting each month is at a "Europe friendly" time, so this would would be at [EDITED to correct time] 16:00 UTC (9am PDT, 18:00 CEST). Does this work for everyone?

Sounds good. I won't be able to make it at that time, but if enough other people can I'll look forward to reading the logs.

@daniel Do you mean 16 UTC or is the plan to swap planning and discussion?

@daniel Do you mean 16 UTC or is the plan to swap planning and discussion?

Gah. You are right, sorry. I'll edit.

@Krinkle, If you want to change the description like that, I think you should include a definition of what exactly you mean by "installation". Otherwise this might come off as an attempt to obfuscate the obvious drawbacks of using only extension.json without composer.json. I think most people would understand downloading an extension and putting the wfLoadExtension into LocalSettings as "installation".

Edit: ... or in fact the obvious drawbacks of this hybrid solution

As I understand it, this proposal would introduce a third download mechanism for extensions. (after Git, and tarballs/ExtensionDistributor).

Comparing how they would work:

RequirementsInitial installUpdateSwitch version
TarballsDownload from mediawiki.org GUI and unpack where neededReplace directoryReplace directory
GitCLI access, Gitgit clone ext-url ext-versiongit pullgit checkout
ComposerCLI access, Composer + Gitedit composer-local.json and run composer updatecomposer updateedit composer-local.json and run composer update

In terms of installilng / managing extensions with MW. They all require the same: Edit LocalSettings.php and add wfLoadExtension() for each extension. Correct?

In terms of relationships between extensions, the RFC says Compooser will not be used to track dependencies between extensions. This would mean dependency-extensions would also need to explicitly mentioned in composer-local.json. Correct?

From the task description:

Advantages:

  • Centralized vendor/ directory: Extensions can store their dependencies in a centralized directory, rather than having them in their respective sub-directory.

This is already the case today. The exception is if someone downloaded extensions using the tarball method (which bundles vendor locally, this RFC doesn't change that, and that's also really hard to avoid in general).

If you run composer locally (docs) then afaik we only have one documented best practice, and that is to run composer update from the central directory (with wildcard from composer-local). Not doing so is afaik unsupported and would cause all sorts of issues (in addition to being a lot of work), because it means Composer doesn't have a full picture ands thus can't resolve dependency ranges very well, race conditions of who wins the autoload include etc.

Composer CLI access, Composer + Git

Git is not strictly needed, composer-local.json determines everything else (well, other than MediaWiki itself). It's a single file, so all kind of naive versioning strategies could work reasonably well, or someone could write a CLI or web tool for generating it (so you can click together what extensions you want on a rich web interface with descriptions, recommendations etc).

CLI access is also not strictly needed, there's stuff like Composercat (whether that's a good idea is another question).

I guess the main weakness is that you'll still need either tarballs or git for installing MediaWiki itself. It would be great to make MediaWiki installable via Composer, but that's a wholly different level of effort, and this RfC has wisely chosen not to concern itself with that.

Composer CLI access, Composer + Git

Git is not strictly needed, composer-local.json determines everything else […]

Users don't need to interact with Git commands when using Composer, however Composer itself requires Git afaik to be able to perform all its expected functions (e.g. not only tagged releases, but also dev-master, dev-REL1_x). In particular because WMF-developed extensions do not tag releases for anything other than core.

Users don't need to interact with Git commands when using Composer, however Composer itself requires Git afaik to be able to perform all its expected functions (e.g. not only tagged releases, but also dev-master, dev-REL1_x). In particular because WMF-developed extensions do not tag releases for anything other than core.

I don't think this is strictly the case -- from https://getcomposer.org/doc/05-repositories.md#git-alternatives it appears that composer can download github-hosted files as ZIP tarballs using the github API. Since WMF-developed extensions are mirrored on github, that means that (in theory, I haven't tested this) you ought to be able to pull zips corresponding to release branches of WMF extensions without having git installed on your host. In any case, I think git ought to be considered a dependency of composer, not of this download mechanism: there are plenty of other ways you might be able to use composer to download extensions independent of git. We shouldn't confuse "how WMF would use composer to download WMF extensions (if WMF were to do so)" with "how third-parties might use composer to download third-party extensions"; the latter certainly doesn't *require* git.

I'd also like to recommend (mirroring what @CCicalese_WMF said in the RFC meeting) that we avoid using the term "install" in the description of this RFC, as potentially confusing w/ multiple meanings. "Download", "unpack", and "enable" seem to cover all the steps involved and are less ambiguous.

There is an implicit software dependency for this RFC that I worry most folks have not thought about or understood. The composer-merge-plugin software which is currently required by mediawiki/core.git's composer.json only exists to make Composer management of extensions and skins possible. This software was written by me (@bd808) in the Foundation's FY2014/2015 so that we could start using composer.json to manage PHP library dependencies for MediaWiki core without destroying the possibility of using Composer to manage other software in a given MediaWiki deployment. Composer-merge-plugin is functionally abandonware today. No Wikimedia Foundation team is actively claiming "ownership" of the project. If this RFC were declined and T249573: Remove support for extensions requiring a MediaWiki version via Composer were accepted, we could remove the dependency on composer-merge-plugin from mediawiki/core.git.

If this RFC is accepted, in my opinion, there must be a Wikimedia Foundation team or long term stable MediaWiki technical community group established to be the "owner" of composer-merge-plugin. This is especially topical at the moment as Composer is preparing for an API breaking 2.0 release upstream and composer-merge-plugin needs active coding and testing to ensure compatibility with those upstream changes.

Unless I'm missing something, composer-merge-plugin is currently the only way cleanly and sanely set up extensions in a development environment. @bd808 is there an alternative to merge-plugin for that use case? Running composer for each extension only works if there are no version conflicts...

Unless I'm missing something, composer-merge-plugin is currently the only way cleanly and sanely set up extensions in a development environment. @bd808 is there an alternative to merge-plugin for that use case? Running composer for each extension only works if there are no version conflicts...

It only works via composer-merge-plugin if there are no version conflicts as well. I'm not saying that the tool is not useful, I'm saying that it is abandonware today and this RFC is codifying the need for it to exist perpetually.

It only works via composer-merge-plugin if there are no version conflicts as well. I'm not saying that the tool is not useful, I'm saying that it is abandonware today and this RFC is codifying the need for it to exist perpetually.

Unless I'm misunderstanding the RFC, I believe it is saying that the plugin would no longer be necessary as composer would be used to download the extension itself (and all it's dependencies).

No, the plugin would still be necessary, as people would need to be able to add extensions to their composer.local.json.

That said, even if this RFC is declined, the Composer Merge plugin is still required because there still needs to be a way to load extensions' dependencies. Because of that, I don't see that the lack of maintenance for the merge plugin is necessarily a stopper for this RFC (although I totally agree that it'd be good if there was some clarity around who maintains it).

I guess an other way to do it would be to remove composer.json from core (move it to composer.dist.json or something) and let wiki admins make their own composer.json. No merge plugin required, but other things would have to change.

Unless I'm misunderstanding the RFC, I believe it is saying that the plugin would no longer be necessary as composer would be used to download the extension itself (and all it's dependencies).

Without composer-merge-plugin, a local diff to mediawiki/core.git's composer.json file would be required to composer install an extenstion. That untracked state will be lost if a new tarball is unpacked over the existing state in the case of a tarball based deploy. With a git based deploy such state could be tracked as a local commit that is rebased on top a a fresh pull, but that workflow will also be awkward.

Use of composer require ... to automate both the composer.json edit and the install in one action is not covered by this RFC currently and would not be corrected by the presence of composer-merge-plugin. That could be seen as a deficiency in the current discussion as it not addressed at all. I will leave it to the group to decide if the desired outcome of this RFC is actually a fully documented and tested workflow or simply retention of the current functionality that was proposed for removal in T249573: Remove support for extensions requiring a MediaWiki version via Composer.

@dbarratt, your own proposal in T166956: Cannot use Composer's CLI to manage a project's dependencies would be needed to cleanly allow both outcomes without helper software such as composer-merge-plugin.

@bd808 I see. apologies for my confusion.

If this RFC is accepted, in my opinion, there must be a Wikimedia Foundation team or long term stable MediaWiki technical community group established to be the "owner" of composer-merge-plugin.

Is there a phab task or discussion somewhere that is tracking that possibility? It's a fairly widely used plugin so if we are not going to keep using it, we should probably try to find a new maintainer.

Composer-based extension management would actually be fairly easy without composer-merge-plugin:

  • move MediaWiki's composer.json out of the way (rename to composer.dist.json, for example)
  • create a Packagist module for MediaWiki core which contains nothing other than core's composer.json file
  • people who install MediaWiki via Composer would have to add that module, alongside their extensions and skins.

(Which is basically a lightweight / hacky version of T166956: Cannot use Composer's CLI to manage a project's dependencies.)

But it's a moot point, because we need to keep composer-merge-plugin alive for git-based installs anyway.

This is especially topical at the moment as Composer is preparing for an API breaking 2.0 release upstream and composer-merge-plugin needs active coding and testing to ensure compatibility with those upstream changes.

The task for that is T248908: composer-merge-plugin support for composer 2.0, FWIW.

I think moving forward with composer and endorsing it as a supported way to distribute and install extensions is a mistake. It feels like we're taking this half-baked idea that just keeps dropping individual components (can't use composer.json in core, moved to composer-merge-plugin/composer.local.json; can't load entrypoints, now dropping that). I think composer is an adequate development tool, but I don't think it meets our needs for extensions. I also think this process is generally backwards: we should be clearly documenting what our needs and wants are (T118188), and then evaluating different solutions against that, rather than picking composer as the solution and retroactively building requirements around that.

I break down extension management into 3 areas: discovery, installing, upgrading.

Discovery:

Pros:

  • packagist is something that already exists, that we can use
  • packagist supports any Git repo that is accessible to the Internet

Cons:

  • Packagist/composer are English-only, and don't support any localization.
  • Packagist is filled with plenty of other PHP stuff, so you need to first filter against MediaWiki things before looking for what you want.
  • Packagist is hosted by a 3rd party with an unfavorable privacy policy (uses Google Analytics)

Installing:

Pros:

  • MediaWiki developers and sysadmins who use composer are already familiar with it.
  • There's plenty of documentation out there for people who have trouble with composer
  • Composer can automatically resolve library dependencies, if they are compatible.

Cons:

  • Composer is command-line only, leaving behind people who use the web installer or aren't that comfortable with CLI
  • Composer introduces multiple third-parties into the integrity verification pipeline, compared to downloading from Wikimedia Gerrit/tarball
  • It's really easy to use the wrong way, regardless of how experienced you are (see: --no-dev and the PHPUnit RCE affecting prominent wikis run by very experienced MW developers)

Upgrading:

Pros:

  • If everything is compatible, it's able to update everything at once, with little interaction needed

Cons:

  • ..? (I'm sure there are some, I can't think any right now)

Based on the way I read the proposal, the proposed usage of composer is down to:

  • a registry
  • automated install tool based on list of names
  • automated updating based on list of names
  • automated fetching and set up of composer dependencies

Am I missing anything? At that point, I think we're better off writing our own dependency manager thing (as I suggested/prototyped before). Even one of the multi-git wrappers would probably be preferable. IMO composer's strength is in it's dependency resolver and semver, but most MediaWiki extension dependency trees are at most one level deep, and rarely conflict with each other (since we expect master always works). Most MediaWiki extensions don't have composer dependencies, so I think optimizing for that case is wrong.

I think (hope?) all of us still dream of the day when extension management is as simple as it is in WordPress, but I fear that going down this composer road is just going to make that dream even more of a pain to achieve.

Technical problems with composer extensions:

  • We currently autoload composer late, for the purpose of supporting extensions, which causes problems with using library code in MediaWiki itself (see https://lists.wikimedia.org/pipermail/wikitech-l/2019-June/092233.html, which has been worked around for now)
  • Having an extension dependency in composer.json messes up the normal CI flow of composer install && composer test because there's now an unexpected ./extensions/Foo directory. This is ususally worked around by teaching each tool to ignore the non-standard extensions directory.

These are off the top of my head, there are definitely more that we've worked around over the years. None are unfixable, but I don't think this is as straightforward as "composer works great, let's codify it".

Surely there are better options, but they have yet to be implemented (cf. @Tgr's comment above). So composer seems to be a viable way to go, it has a solid usage basis and a set of MediaWiki developers willing to maintain it.

See T118188#2613011 / https://gerrit.wikimedia.org/r/c/mediawiki/core/+/308891 - it's hard to move forward when you're met with silence...

Most MediaWiki extensions don't have composer dependencies, so I think optimizing for that case is wrong.

Many of them do, and I think even more would do, if it weren't that cumbersome to add (required non-dev) composer dependencies to WMF deployed extensions.

Regardless whether we go with another solution or composer, I think the solution should include way of installing composer deps and alert about incompatible requirements (ideally across core, extensions and their composer deps).

Most MediaWiki extensions don't have composer dependencies, so I think optimizing for that case is wrong.

Many of them do, and I think even more would do, if it weren't that cumbersome to add (required non-dev) composer dependencies to WMF deployed extensions.

My hastily written script (P11646) found 7% (59/833) of extensions in Wikimedia Gerrit have composer dependencies - I think that's pretty minimal. I do agree that it's likely some more would have composer dependencies if it were straightforward to do so,

Regardless whether we go with another solution or composer, I think the solution should include way of installing composer deps and alert about incompatible requirements (ideally across core, extensions and their composer deps).

Agreed.

Surely there are better options, but they have yet to be implemented (cf. @Tgr's comment above). So composer seems to be a viable way to go, it has a solid usage basis and a set of MediaWiki developers willing to maintain it.

See T118188#2613011 / https://gerrit.wikimedia.org/r/c/mediawiki/core/+/308891 - it's hard to move forward when you're met with silence...

Putting my money where my mouth is, here's another prototype: P11658 (Python because I'm faster in it than PHP still, but it should be trivially translatable). It supports git (using REL1_XX branches) and extdist and having composer dependencies. We could probably figure out a scheme for semver git tags too. It figures out what the locally intalled version is, what the latest version available for download is, and updates if necessary. If there are composer dependencies, it'll trigger composer update. In theory if it went wrong we'd be able to rollback as well. Here's an example of what it looks like if you're moving from REL1_31 to REL1_34.

$ ./mwext.py update
Getting the latest version of TemplateStyles
/home/user/projects/mwext-prototype/test/extensions/TemplateStyles needs an update
Getting the latest version of TemplateStyles
[extdist]: Updating /home/user/projects/mwext-prototype/test/extensions/TemplateStyles...
[extdist] Downloading https://extdist.wmflabs.org/dist/extensions/TemplateStyles-REL1_34-c4d6f25.tar.gz
[extdist] Successfully updated /home/user/projects/mwext-prototype/test/extensions/TemplateStyles
Getting the latest version of AbuseFilter
/home/user/projects/mwext-prototype/test/extensions/AbuseFilter needs an update
Getting the latest version of AbuseFilter
[git]: Updating /home/user/projects/mwext-prototype/test/extensions/AbuseFilter...
Previous HEAD position was 7b66a2c8 SECURITY: Unbreak blocks shorter than one hour
HEAD is now at 3c2035dd SECURITY: Unbreak blocks shorter than one hour
[git] Successfully updated /home/user/projects/mwext-prototype/test/extensions/AbuseFilter
Triggering composer update...
Loading composer repositories with package information
Updating dependencies
Package wikimedia/password-blacklist is abandoned, you should avoid using it. Use wikimedia/common-passwords instead.
Generating optimized autoload files

TL;DR: Perfect is the enemy of Good Enough.

Developing a custom extension manager will require having the WMF commit to maintaining it in the future. The approach proposed in this RFC would require continued maintenance of composer-merge-plugin - a component that, as noted in T250406#6233269, is also necessary for other reasons. Given how difficult it has been to find someone at the WMF to maintain that code, I am skeptical of the long-term maintenance of some yet-to-be developed code.

But, let's back up a second.

The goal of this RFC is simply to codify the best practices and anti-patterns of using composer to download extensions--something developers are already doing. This has been an open area of concern for many years, and an alternative approach to composer has not materialized.

Perhaps in the future, such an approach will indeed be developed, but in the meantime, documentation is needed for the best approach in the current situation.

If another solution with WMF maintenance buy-in appears, that's great, but we (third party users like SMW devs and Hallo Welt!) need to codify what already exists.

TL;DR: Perfect is the enemy of Good Enough.

Developing a custom extension manager will require having the WMF commit to maintaining it in the future. The approach proposed in this RFC would require continued maintenance of composer-merge-plugin - a component that, as noted in T250406#6233269, is also necessary for other reasons. Given how difficult it has been to find someone at the WMF to maintain that code, I am skeptical of the long-term maintenance of some yet-to-be developed code.

A properly developed extension manager would actually be used by Wikimedia production, therefore getting the maintenance and love it needs. Adopting composer would suffer from the same problem you explain - Wikimedia production isn't going to adopt it. That's fundamentally the same state composer-merge-plugin is in, it's used for a specific featureset that CI uses, and that's just about what's maintained.

But, let's back up a second.

The goal of this RFC is simply to codify the best practices and anti-patterns of using composer to download extensions--something developers are already doing.

I fundamentally disagree with using composer to manage extensions as an officially supported method by MediaWiki. There are significant unaddressed flaws with this method (partial list in my previous comments) that are being ignored because the people using them don't perceive them as problems, even though they are or have gotten frustrated enough to work around them.

This has been an open area of concern for many years, and an alternative approach to composer has not materialized.

Perhaps in the future, such an approach will indeed be developed, but in the meantime, documentation is needed for the best approach in the current situation.

If another solution with WMF maintenance buy-in appears, that's great, but we (third party users like SMW devs and Hallo Welt!) need to codify what already exists.

I would posit that the constant pushing of a flawed composer-based solution has taken any remaining energy from potential alternatives, but I don't want to get into that - moving forward is more important.

In any case, if the goal is just documentation...just document it. https://www.mediawiki.org/wiki/Composer/For_extensions exists, so update it. As long as people know what they're getting into (the warning banner at the top of the page), extension maintainers can distribute their extension however they want. It just won't be officially supported by MediaWiki, with the potential for breaking changes caused by subtle initialization changes, etc.

  • We currently autoload composer late, for the purpose of supporting extensions, which causes problems with using library code in MediaWiki itself [...]
  • Having an extension dependency in composer.json messes up the normal CI flow of composer install && composer test because there's now an unexpected ./extensions/Foo directory. [...]

This RFC wants composer to be used for installation and update of MediaWiki extensions and their respective libraries. By this RFC an extension must not have any "static" files be included by the autoloader. Not to enable itself, not to include any default settings or manipulate/access the global state. Not even the PSR-4 autoloader of composer should be used. That's what the ExtensionRegistry is for. The RFC states that "inter-extension-dependencies" are only to be declared through ExtensionRegistry. If the "requirements" of this RFC get implemented, the issues mentioned above should be prevented. Simple CI tests could be introduced that make sure those rules are being followed.

The important thing is, that these practices get to be known and followed by extension developers. Therefore they should be "official". Third-party developers look at what the WMF recommends and what the WMF actually does in their extensions.
The only good way to encourage developers to use libraries is to also allow them to install the extensions using composer. Thus having library dependencies and constraints properly checked and the code installed in a central location.

I just want to get over "you must not use composer for anything else than CI tests" and come to a commonly accepted "using composer for installation is okay if you follow these simple rules" situation. But this requires a little support by the WMF. And that's what this RFC is about.

I would like to see MediaWiki core, libraries, extensions and skins installable by composer -- just like you can do with Drupal (introduced in Drupal Core 8.8 and mentioned earlier in this RFC). "But wiki extensions don't use semantic versioning." Drupal modules (mostly) don't follow semantic versioning, but there's a shim to accomodate that. End users aren't forced to use composer - because it's not the only way to download/install/manage Drupal. In fact, it's not even mentioned on the project download page. The point is, I wholeheartedly support this RFC in terms of improving and clarifying composer support in the MediaWiki ecosystem.

As an extension developer, I want to be able to user composer for my extension dependencies, and I want those to be recognized by MediaWiki. At the present time, we need to declare dependencies in extension.json and composer.json, (and in zull/parameter_functions.py for CI)

The warning at the top of https://www.mediawiki.org/wiki/Composer/For_extensions seems heavy-handed. The declined RFC was from 2013! The emphasis on "is likely to be removed" needs a citation.

I would like to see MediaWiki core, libraries, extensions and skins installable by composer -- just like you can do with Drupal (introduced in Drupal Core 8.8 and mentioned earlier in this RFC).

Also, if you "require" an extension with Composer in Drupal, it does not enable that extension. I think the same should happen for MediaWiki.

Also, if you "require" an extension with Composer in Drupal, it does not enable that extension. I think the same should happen for MediaWiki.

That is the current state: If an extension is required it is not enabled. We've worked with extension developers to ensure that it isn't activated automatically.

Hello everyone. What is the next step here?

TechCom reviewed this proposal again last night. We see consensus from the technical perspective, and normally we would move this to P4. However, it is a strategic direction with unclear resourcing, so we’re reaching out to managers. Once we establish that teams are empowered to spend time on applying and maintaining this approach, we can move forward.

Those of us working on this have updated Composer/For_extension_developers and Composer/For_extensions in anticipation of the resolution of this.

Those of us working on this have updated Composer/For_extension_developers and Composer/For_extensions in anticipation of the resolution of this.

Thanks mark!

It is my understanding that this RFC effectively asks the WMF to release all extensions she maintains on a composer repository (packagist or self-hosted) on a regular bases (along with core releases, presumably). Did I get that right?

I have been talking to @MNadrofsky and @WDoranWMF about getting commitment for that. Since extensions are owned by various teams, this may take a while.

It is my understanding that this RFC effectively asks the WMF to release all extensions she maintains on a composer repository (packagist or self-hosted) on a regular bases (along with core releases, presumably). Did I get that right?

Yes. This work could be done by volunteers if the RFC is accepted, but it is best if the WMF does the work.

It is my understanding that this RFC effectively asks the WMF to release all extensions she maintains on a composer repository (packagist or self-hosted) on a regular bases (along with core releases, presumably). Did I get that right?

Yes. This work could be done by volunteers if the RFC is accepted, but it is best if the WMF does the work.

What work would the be? Most WMF maintained extensions we don't actually version properly (at least as far as semver... We're getting better about updating the extension.json requires though), nor do we do tagged releases...

Or is the existence of the REL1_* branches into packagist (or similar) just enough?

Per https://packagist.org/about#managing-package-versions:

Branches will automatically appear as "dev" versions that are easily installable by anyone that wants to try your library's latest and greatest, but that does not mean you should not tag releases.

So, the REL1_* branches would be sufficient, if the preferred semantically versioned tags are not available.

Yes. This work could be done by volunteers if the RFC is accepted, but it is best if the WMF does the work.

Even if volunteers did the work, the teams owning the extensions would need to at least be in the loop and perhaps provide reviews.
I'm not really worried about the work, it's not a lot. The issue is more about awareness and mental load.

I've updated the description to remove the requirements that would increase the workload for the WMF. Hopefully this will mean it can be fully implemented.

As Cindy pointed out, SemVer is not a requirement. REL-branches will do. It would be great if composer.json files could have required information like name and type right away, but to to keep things simple, it is sufficient if patches that are adding those are accepted by the WMF. If they are not in new WMF extension, volunteers can still add them later. No need for WMF staff to keep in mind. Also registration on packagist.org is not required. A package registry (e.g. packages.mwstake.org) could be set up easily once the repos feature good composer.json files.

I am currently hosting a Composer repository mirroring the MediaWiki extension directory (mostly as a proof of concept to show that installing extensions with Composer can work and can be useful).
I would of course be happy to replace it with Packagist or an official MediaWiki repository.

@Rudloff This is awesome! The BlueSpice team maintains something very similar (https://packages.bluespice.com), but we only have a small selection of packages. Is the script that you use to create the package catalogue available to the public?

@Osnard The code is available here.
There is no documentation but feel free to open an issue on the repository if you have questions.

Is there any progress happening in this question?
I'd be very happy to see better and "official" support for composer to manage mediawiki installations.
I'm not involved with wikimedia and don't know the community very well but I'd give my "outside" perspective: I run a mediawiki website with a number of extensions and customizations for some years. Also I run Drupal9 websites and learned to manage them with composer. It was a learning curve at the beginning, but now I'm very happy with it: It's effective and fast, I can easily put it in a git repo and have revisions of my installation... In case something really goes wrong I can go back within minutes - I find this workflow quite beautiful.
I can observe that I'm quite fast with updates with my Drupal websites and the reason is this workflow.

Whereas with mediawiki... well I switched from tarballs (touching the website only once in two years, taking a day to update from one LTS to the next LTS) to managing everything with git. But still updating is painful. Working with git submodules just isn't elegant, my last update took me hours - I really wonder: how do others do that, am I missing something?

I can observe that I'm very hesitant to touch my mediawiki installation, rather using the workaround from a security advisory than trying the painful update process.
I constantly feel a bit bad about that and I thought "Wouldn't it be amazing if mediawiki would also use composer?" - investigating that questions I found this RFC.
My feeling is that there isn't the capacity here to develop and maintain a custom system (I guess there is also much less mediawiki websites out there than Drupal websites). Also it would make it even harder for people like me because it would require to get familiar with yet another tool which would be mediawiki-specific.
Composer instead is quite widespread and there is many others who are investing in it. After all, Drupal has essentially a similar architecture where you manage everything with composer but still need to "activate" extensions (similar to wfLoadExtension() in mediawiki) and is successfully using composer, I think mediawiki would benefit from doing that as well.

My feeling is that there isn't the capacity here to develop and maintain a custom system (I guess there is also much less mediawiki websites out there than Drupal websites). Also it would make it even harder for people like me because it would require to get familiar with yet another tool which would be mediawiki-specific.
Composer instead is quite widespread and there is many others who are investing in it. After all, Drupal has essentially a similar architecture where you manage everything with composer but still need to "activate" extensions (similar to wfLoadExtension() in mediawiki) and is successfully using composer, I think mediawiki would benefit from doing that as well.

@Holybiber: thank you, I could not have said it better myself.

As I commented there, I agree with this from @Krinkle:

you must remove the use of "extensions/*/composer.json" and instead fully embrace the "Composer way", i.e. use "require" to list out each extension you want to install

Using * was not my idea, it is in the manual

https://www.mediawiki.org/wiki/Composer#Using_composer-merge-plugin
To enable Composer to discover and process the composer.json files that may be included in any and all of your locally installed extensions, add something like this to $IP/composer.local.json:

{
    "extra": {
        "merge-plugin": {
            "include": [
                "extensions/*/composer.json",
                "skins/*/composer.json"
            ]
        }
    }
}

I understand that it is almost impossible to find a consensual solution here. However, I think the pull requests to composer or its merge plugin improve the lives of some people without making it worth for others. Manually creating and maintaining composer.local.json for more complex non-wmf setups is at least more effort than patching the merge-plugin or installing a custom composer build.

https://github.com/wikimedia/composer-merge-plugin/pull/251 describes a current problem with extensions using composer.

I don't think this has anything to do with using Composer for extension management. AIUI the alleged issue is that if you use composer-merge-plugin, the same Composer library appears multiple times as a dependency, and it has an autoload.files field in its composer.json (ie. it tells Composer to load some file unconditionally, typically because it has global functions in it), the file will be loaded multiple times, causing a PHP error. That does sound like a bug in composer-merge-plugin (or Composer, arguably, but the maintainer doesn't seem interested in fixing on their side), regardless of what the plugin is used for.

[...] However, I think the pull requests to composer or its merge plugin improve the lives of some people without making it worth for others. [...]

I agree. As long as this change does not affect other (currently working) configurations it should be merged.

Nevermind, I misread the bug report. https://github.com/wikimedia/composer-merge-plugin/issues/247 is clearer. The conflict is between the same extension first required via Composer's normal require mechanism and second via composer-merge-plugin.

There doesn't seem to be any easy way to avoid that from happening - if some of your extensions are installed via Composer, and others aren't, you need both the requires for the first group and composer-merge-plugin for the second, and there is no way to add exceptions to a glob expression.