Page MenuHomePhabricator

RFC: Hybrid extension management
Open, MediumPublic

Tokens
"Like" token, awarded by Akuckartz."Like" token, awarded by MarkAHershberger."Like" token, awarded by CCicalese_WMF."Love" token, awarded by dbarratt."Like" token, awarded by RHeigl."Like" token, awarded by freephile."Like" token, awarded by Kghbln."Like" token, awarded by JeroenDeDauw.
Assigned To
None
Authored By
MarkAHershberger
Apr 16 2020, 4:41 PM

Description

Motivation

Over the last years there have been several attempts to use the Composer "package manager" as a way to manage MediaWiki extensions. At the moment there are already a lot of extensions that support Composer as a way of installing and updating.

The main concerns about using Composer as an extension-management-tool is that Composer "is for managing libraries, not extensions" and that extensions would bypass wfLoadExtension(). This second part was a special concern for multiple wikis hosted from the same codebase (i.e. wikifarms).

Requirements

The responsibility of "managing" extensions should be split up between Composer and MediaWiki's ExtensionRegistry. It would become a "hybrid system".

  • Composer can be used to download and update extensions and their libraries, but does not enable the extension
  • ExtensionRegistry is responsible to check for inter-extension-dependencies as well as for version constraints (e.g. to the core MediaWiki version)

This means that an incompatible or unsatisfied extension may be downloaded by Composer but ExtensionRegistry would not install it.

  • composer.json => Evaluated when downloading or updating source code.
  • extension.json => Evaluated when running MW (installer/updater, and page views).
Advantages
  • Acceptance by the community: Composer is already widely used. Other open source communities also use it for "module-management", not just library-management
  • Centralized vendor/ directory: Extensions can store their dependencies in a centralized directory, rather than having them in their respective sub-directory. Therefore multiple different versions of library code can be prevented. Also, Special:ExtensionDistributior seems not to handle libraries at all at the moment (T215713)
  • Support for "SemVer" and "release branch compatibility model": Composer can use SemVer versions as well as release branches ("dev-REL1_*"). This makes it easy for a user to setup a reliable update mechanism (e.g. by "subscribing" to the "REL1_*" branches only).

Exploration

The current situation can easily be improved by some minor policy and software changes:

Changes on the software
  • ExtensionRegistry would throw an exception if requirements specified in extension.json were not met. Separate exception handling code (like SMW's SetupCheck) would be used for exceptions that happen during these early checks to ensure that administrators get better information than the WSOD.
Changes on the policy
  • Extensions must not enable themselves automatically via installation by Composer (e.g. by autoload.files)
  • Extensions must only declare dependencies to libraries in the composer.json. All dependencies to MediaWiki core versions and inter-extension-dependencies must be declared in the extension.json (By this, a virtual MediaWiki package is not required).
  • The WMF should not just allow, but encourage the use of "packagist-compatible" (must have fields like name, type and extra.installer-name) composer.json files in the extension repositories. This should be the case for all actively supported branches (LTS) of the WMF owned extensions.
Potential action items
  • Add composer-guidelines to the official MediaWiki extension developer resources
  • Make all composer.json files in WMF extension/skin repos packagist-compatible
  • Either register all WMF extension/skin repos with packagist.org or setup a dedicated package-registry (e.g. packages.mediawiki.org)
  • Improve ExtensionRegistry dependency error handling (to be discussed)

See also

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Reedy added a comment.Apr 16 2020, 6:34 PM

@Krinkle, I'm confused by your example. Could you rephrase it?

E.g. given an ecosystem that doesn't use Composer for extensions, what would this RFC make simpler, cheaper or possible? (as opposed to just different).

But, the ecosystem does use composer for extensions. This RFC just tries to bring the RFC-given-guidance into line with reality.

Not in any sort of official supported/approved way - see the docs. The RFC was declined, and then later a patch created and merged without any associated task, or any real discussion.

Yes, we do use (and support) composer for extensions composer/packagist dependancies (ie libraries), but not for anything else

Not in any sort of official supported/approved way - see the docs. The RFC was declined, and then later a patch created and merged without any associated task, or any real discussion.

Ok, so we're at a point now where it is not officially supported (but composer is, in fact, used this way) and this RFC and the one you're working on (T249573) aim to resolve this issue.

(If that isn't a good summary, please let me know.)

While I understand that the Foundation is not likely to adopt composer to install extensions for itself. the fact is that there are plenty of other users of MediaWiki and there are extensions that use composer that are not intended for WMF's use.

Accepting this RFC would acknowledge that MW has grown in ways that the WMF did not plan for or foresee.

I note that one of the main reason for declining the earlier proposal for composer based extension management was that it enabled extensions directly. This seems to have been addressed.

There seems to be a strong desire for using composer this way for third party installs. If there is a simple and clean way to allow extensions to support this, I see no reason to prohibit a-priori.

What I don't quite see is what this RFC is proposing. Do we need to change something in MediaWiki to make this possible? Or is this just about establishing best practices?

Reedy added a comment.Apr 16 2020, 6:58 PM

Yeah, I'm not potentially against having something that the WMF could use, but doesn't (I don't know exactly how this looks. Whether it's what we've already got, or it's something else). And has someone to actually support/maintain it.

We all know MW config management, and to a similar extent, skin/extension management sucks, and I doubt anyone is disputing that.

My task (following on from MaxSem's patch) at T249573 was to try and normalise stuff and remove the mostly unsupported method. We get a few people a week that come on IRC and are confused why they can only install some extensions via composer etc.

The inconsistency is the worst part. Documentation should be made to reflect reality, in some way... Even if the case is that only some extensions can be installed like that.

Though, this kinda irks me to some extent. Having installation method A for some subset of extensions, installation method B for others, and C where they support both A and B... Well, frankly, is just a mess and leads to much more confusion :)

Reedy added a comment.Apr 16 2020, 7:01 PM

^ Basically, if we're going to do this.... Let's do it properly

Note that @Osnard was already working on this RFC before we saw T249573. When we saw that, we knew we had to put our not-quite-finished RFC up for some initial review. I do not have time right now to flesh out the details and @Osnard is unavailable until next Monday at the earliest.

Please do post issues that need to be resolved and we'll have more time to work on this in the next week.

Though, this kinda irks me to some extent. Having installation method A for some subset of extensions, installation method B for others, and C where they support both A and B... Well, frankly, is just a mess and leads to much more confusion :)

One more note: When we began formulating this RFC on mwstake.org, we wanted to avoid what we saw as a political nightmare of switching from "Composer can't be used for installing extensions" to "Composer is the ONE TRUE WAY to install extensions" and the current formulation reflects that.

With @Osnard's efforts, Hallo Welt! has been able to create things like https://packages.bluespice.com/ that allow them to install (all?) MediaWiki extensions using Composer.

Krinkle added a comment.EditedApr 17 2020, 12:44 AM

@Krinkle, I'm confused by your example. Could you rephrase it?

E.g. given an ecosystem that doesn't use Composer for extensions, what would this RFC make simpler, cheaper or possible? (as opposed to just different).

But, the ecosystem does use composer for extensions. This RFC just tries to bring the RFC-given-guidance into line with reality.

I was referring to WMFs deployment (and various third parties) as example ecosystems where things currently work without Composer being used for extensions.

[…] we wanted to avoid what we saw as a political nightmare of switching from "Composer can't be used for installing extensions" to "Composer is the ONE TRUE WAY to install extensions"

I understand that this RFC is not intending to require use of Composer. And things can't always be one-size-fits-all, that's okay.

But, just because we can do things in two different ways, doesn't neccecarily mean that we should. So my question remains: What do MediaWiki developers and/or future site administrators gain by having an alternative option to do what appears to be the same thing? Is there some fundamental benefit that others might enjoy from this option? The answer may very well be Yes. There's also the matter of balancing that against the indefinite and accumulative cost of supporting and documenting MediaWiki around that and the potential confusions, mistakes or limitations that stem from supporting both.

Before we get to that, though, I'd like you to start by writing down those potential benefits. I fear that otherwise it will be difficult for others to meaningfully participate.

I think the main benefit is ease of installing and upgrading extensions.

Currently, a wiki sysadmin has to take separate action for every extension they have installed:

  • using ExtensionDistributor: check each extension manually, first on their wiki and then on MediaWiki.org to see if there's a newer version, then download, extract, delete the old one, and install the new one.
  • using Git clones:
    • for extensions with 'rel' compatibility policy: change to the right branch, then Git pull.
    • for extensions with 'master' compatibility: git pull

With Composer, this all just becomes composer update --no-dev -o.

Of course, we will still support installations without shell access, so there's always going to be multiple ways of doing some of this (not to mention the Git submodules of selected extensions in core).

Tgr added a subscriber: Tgr.Apr 19 2020, 5:27 PM

So basically this RfC recommends 1) merging the patch Reedy mentioned, 2) changing ExtensionRegistry so that when an extension does not meet requirements, it's skipped quietly instead of throwing an error? As far as I can tell, everything else described there is already the status quo.

Tgr added a comment.Apr 19 2020, 5:33 PM

With Composer, this all just becomes composer update --no-dev -o.

Except you probably won't end up with the right extension version; that will just install that latest version that still matches whatever constraints you have manually specified in your composer.local.json, and throw an error if that version is not compatible with your MediaWiki core.

Except you probably won't end up with the right extension version;

You will if extension authors take care to bump the major version when introducing breaking changes (i.e. when dropping support for an older MW release). There's currently no policy about what version numbers mean for extensions, but if they're installed with Composer then they'll have to follow SemVer.

This is maybe another point of friction with this RFC, but I think it's a worthwhile one: properly versioned (and tagged) releases are useful in general, and not just for Composer.

Tgr added a comment.Apr 21 2020, 1:45 PM

If MyExtension v1.0 is compatible with MW 1.31+ and v2.0 is compatible with MW1.35+ and you have downloaded MW 1.31, added "mediawiki/myextension": "*" to composer.json and ran composer update, Composer won't try to figure out that 1.0 is the last version that can be used in that situation; it will just give you an error about how the requirements could not be resolved.

Also it works poorly with the release branch compatibility model, which is the most convenient way for an extension author hosting their code on Gerrit to manage compatibility.

There are some really good points here, which made me to reconsider my stance on this.

Most extensions do not follow semantic versioning. I bet 90% of them don't do any kind of releases. This is a reason against using supporting composer as install method. Even if we did have everything semantically versioned, having to micro-manage the versions for dozens of extensions is no fun. In my deployments, I mostly stick to REL_XX branches of extensions to match my core version, with exceptions on extensions I maintain for either stable release or master branch, plus explicit versions of a few semantic extensions that do not provide REL-branches. If composer cannot select compatible versions, this is a reason against supporting composer as install method.

Which brings me back to this RfC: I think the scope should be made narrower. I do agree on the following points suggested:

Extensions must not enable themselves automatically via installation by Composer (e.g. by autoload.files)
Extensions must only declare dependencies to libraries in the composer.json. All dependencies to MediaWiki core versions and inter-extension-depenencies must be declared in the extension.json (By this, a virtual MediaWiki package is not required).

These two points solve concrete issues. This could be "RfC: best practices with composer.json and extension.json".

I don't agree with these points, which are about making extensions installable with composer:

The WMF should not just allow, but encourage the use of "packagist-compatible" (must have fields like name, type and extra.installer-name) composer.json files in the extension repositories. This should be the case for all actively supported branches (LTS) of the WMF owned extensions.
MediaWiki's ExtensionRegistry must not break the wiki in case the requirements declared in extension.json (field requires) could not be met. Instead it should leave the extension disabled, log the issue and put out a proper message to the user/admin (e.g. in a section "Loaded but not enabled extensions" on "Special:Version")

As a MediaWiki deployer, I want to have a specification file with list of extensions and skins. With simple command I will get these installed and/or updated. By default it would try to find compatible versions of everything, with overrides possible. And it should be fast too! Given it seems impossible to reach this with composer, I don't see what's the point to make extensions installable through composer. This is a problem, and I think it deserves to be discussed separately. There are hopefully better solutions than composer. E.g. doing it the custom shell scripting way, which does not require the burden of extension developers making releases and registering in packagist.

Not loading an extensions that is specifically asked to be loaded can be both dangerous and create lots of unexpected issues (security extensions, all post-edit etc. actions would not be executed, some content not showing at all). I think failing fast is the best thing here. As much as I hate Semantic MediaWiki's "you must update your database" bringing a wiki down, in most cases it is still the most helpful thing to do. You will notice the issue immediately and can take a correcting action.

Here are some advantages that I see:

Support for "SemVer" and "release branch compatibility model":

It's true, that only very few extension developers use SemVer. Many users of extensions will probably rely on the "release branch compatibility model". They will go to Special:ExtensionDistributor, select their MediaWiki core branch and download the tarball. Or they will manually clone the extension repo and checkout the REL1_XX branch they need.
Something like this can be improved by using composer and specifying dev-REL1_XX as a version constraint. Of course this is not the recommended way to specify a version constraint, but it fits well to the "release branch compatibility model". The BlueSpice MediaWiki distribution uses this (e.g. https://github.com/hallowelt/mediawiki/blob/REL1_31/_bluespice/build/bluespice-free-distribution/composer.json). The user can require multiple extension in his composer.local.json and run composer update at any time. All the extensions will be updated to the latest state of the release branch. No need to visit Special:ExtensionDistributor for each extension or run git pull on each cloned repo.

Centralized vendor/ directory

MediaWiki core comes with a vendor/ directory. When an extension requires a library via composer, Special:ExtensionDistributor will put this library into the vendor/ directory of the extension. This is okay, as ExtensionRegistry has load_composer_autoloader. But when two extensions require the same library it will be added twice to the overall codebase (one time in each extensions vendor/ directory). If those extensions require different versions of that library, there is no way for a user to see this, when installing from ExtensionDistributor-tarball (when installing from a git clone, one would also need to run composer install within the extensions directory after git pull). Installing the extension with composer will put the library into MediaWiki core's vendor/ and will also check if any other extension has colliding requirements.

As an aside

MediaWiki core comes with a vendor/ directory. When an extension requires a library via composer, Special:ExtensionDistributor will put this library into the vendor/ directory of the extension.

I thought that was broken? :)

T215713: Missing dependencies in extension snapshots

Maybe. Wasn't aware of that.

Krinkle added a comment.EditedApr 29 2020, 8:41 PM

Based on today's TechCom triage we have the following questions:

  • Does your proposed outcome require the current MediaWiki-version-for-composer logic to remain in core, or can that be safely removed?
  • Is it likely that proposals for this RFC will need changes in core? Or could this live entirely as an extension and/or set of social conventions?

The motitivation looks good and I think is understandable enough to faccilitate a healthy conversation between stakeholders. Given resourcing is in place as well, I've moved it to Phase 3 for you. See RFC process for details.

Krinkle moved this task from P1: Define to P3: Explore on the TechCom-RFC board.Apr 29 2020, 8:41 PM

Does your proposed outcome require the current MediaWiki-version-for-composer logic to remain in core, or can that be safely removed?

From my point of view it can be removed, as composer will only be used to install, but ExtensionRegistry will be used to to MediaWiki core and inter-extension requirement checks.

Is it likely that proposals for this RFC will need changes in core? Or could this live entirely as an extension and/or set of social conventions?

There will be a minor change in the ExtensionRegistry required. Instead of throwing an Exception in case of a requirements-issue, it should just leave the extension deactivated and provide the user with proper information about the case. E.g. by listing it in a dedicated section on Special:Version. Everything else is conventions and having the WMF acknowledge that people are using composer as an installation method.

Btw.: I am going to update the description of this task soon, to make it more clear what is required in "changes on the software", "changes on the policy" an what are the "advantages" and potential "action points". Asking for patience.

Tgr added a comment.May 1 2020, 2:59 PM

There will be a minor change in the ExtensionRegistry required. Instead of throwing an Exception in case of a requirements-issue, it should just leave the extension deactivated and provide the user with proper information about the case.

As Niklas said, deactivating extensions is not feasible. It could compromise site security, disable anti-abuse features, cause database corruption etc. Maybe there could be a special endpoint which provides compatibility information in a machine-readable way, but the main functionality of the site should be disabled if extensions that are expected to be loaded fail to load.

Given it seems impossible to reach this with composer, I don't see what's the point to make extensions installable through composer. This is a problem, and I think it deserves to be discussed separately. There are hopefully better solutions than composer.

Having a CLI updater seems like a nice middle option for sites which are too big for tarballs (e.g. involve multiple servers, or just want to avoid the vendor compatibility problem Robert mentioned) but too small for the overhead of managing your own git repo of extension submodules and vendor libraries. Composer is not useful for dependency resolution, but if you can generate a spec with exact versions pinned, it is perfectly good for downloading the files. It's a bit slow but most people will happily take a slow updater that actually exists over a fast hypothetical one. So making sure all extensions are composer-installable (which just means having a composer.json and a reference to the MediaWiki-specific Composer customization package that makes them install in extension instead of vendor) is a reasonable goal IMO. I would much prefer to see a proper MediaWiki update script that does its own download handling, but with no one willing to do the work, that's not a good reason to block Composer from being used for that purpose.

daniel added a comment.May 1 2020, 8:03 PM

There will be a minor change in the ExtensionRegistry required. Instead of throwing an Exception in case of a requirements-issue, it should just leave the extension deactivated and provide the user with proper information about the case.

As Niklas said, deactivating extensions is not feasible. It could compromise site security, disable anti-abuse features, cause database corruption etc. Maybe there could be a special endpoint which provides compatibility information in a machine-readable way, but the main functionality of the site should be disabled if extensions that are expected to be loaded fail to load.

I agree that this is a show stopper. Imagine an extension like Lockdown adding a requirement that is not met on some installation, and the extension getting silently disabled due to that. This would immediately drop all restrictions imposed by Lockdown, potentially exposing sensitive information. That's a no go.

I don't understand why this behavior is required for extension installation via composer, though.

This RfC brings a long needed clarification. That's why I strongly support it. There are over 200 extensions out there that can be downloaded using composer, including some heavy-weights such as SMW, BlueSpice, Wikifab and Chameleon skin. Now the RfC basically codifies a best practice and a way forward for a recommended use of composer. Not doing so will not make composer based extensions go away, but leave their developers in a state of insecurity, which is not helpful.

Personally, I have come to appreciate the easiness of updating the codebase of composer-managed MediaWikis, both in development and in productive environments. Surely there are better options, but they have yet to be implemented (cf. @Tgr's comment above). So composer seems to be a viable way to go, it has a solid usage basis and a set of MediaWiki developers willing to maintain it.

It seems to me that the only remaining controversial point here is the silent auto-disabling of extensions. Maybe a good compromise would be to handle version conflicts in a user-friendly way. So the site won't technically break but display an error message telling the maintainer there is a version conflict (and maybe even a hint on how they can resolve it.)

Tgr added a comment.May 18 2020, 10:19 PM

Maybe a good compromise would be to handle version conflicts in a user-friendly way. So the site won't technically break but display an error message telling the maintainer there is a version conflict (and maybe even a hint on how they can resolve it.)

Version checks have to be done before hooks and similar extension points are invoked (otherwise you'll just end up with a "Class XXX not found" exception which is definitely not user-friendly). Short of major refactoring, they have to be done before the ExtensionRegistry::loadFromQueue call, which is near the top of Setup.php. At that point, basic stuff like DB handling, object cache, i18n have not been initialized; even the exception handlers aren't set up. Rendering anything resembling a wiki page is not feasible. Not sure what you mean exactly by the site "technically not breaking" but I doubt this will be possible. Throwing an exception is the best that can be done in that situation.

Maybe MWExceptionRenderer could be split into a simple and a complex part, where the simple does not depend on OutputPage etc, and the simple version could be installed for the early part of Setup.php.

even the exception handlers aren't set up.

It's true, but maybe ExtensionRegistry::queue() could handle things in a similar way to how DBConnectionErrors are handled, i.e. with HTML output that sort of explains things, rather than the usual WSOD. I'm sure bits of MWExceptionRenderer could be refactored to make it possible without being too strange (i.e. using that class not via the exception handler is probably bad).

Osnard updated the task description. (Show Details)May 22 2020, 4:29 PM

I wholeheartedly support this proposal. It clearly explains a position that I have advocated for, for years.

The proposal is similar to what Drupal does:
https://www.drupal.org/docs/develop/using-composer/using-composer-to-install-drupal-and-manage-dependencies

Effectively composer is used at install time, but at runtime the enabling of an extension (and the autoloading of the classes in said extension) is handled by the CMS. This means that an extension can be enabled/disabled from the UI and when it is disabled, the code isn't executed at all (though, does continue to exist in the codebase until it is removed with Composer).

dbarratt updated the task description. (Show Details)May 22 2020, 5:53 PM

[...]
Throwing an exception is the best that can be done in that situation.

Maybe MWExceptionRenderer could be split into a simple and a complex part, where the simple does not depend on OutputPage etc, and the simple version could be installed for the early part of Setup.php.

I think this refactoring would be best.

For what its worth SMW has created SetupCheck which made it possible to display error messages in a much more user-friendly manner when there is a problem with the setup.

Since WMF does not use these code paths, it seems that trying to adopting and adapting this code would be a reasonable thing to do.

I strongly support this RFC. It will serve not only to document best practices for the use of composer for MediaWiki extensions and skins, but will also replace T467 with a policy that better reflects the current recommended state of the practice. I'll repeat my comment from T249573 below for the record.

The situation with respect to use of composer in extensions has changed significantly since T467. I used to be opposed to composer use in extensions myself, but I have been convinced by seeing how composer is currently used in extensions. Today, the use of composer by some third parties to install extensions and maintain their infrastructure is quite impressive.

One of the early objections to composer use had to do with bypassing extension registration and using composer to enable extensions, but even Semantic MediaWiki has changed so that it no longer automatically enables itself. It is important to differentiate between installing an extension, which can supported in many ways, and enabling an extension.

There are many third party MediaWiki extension developers who are using composer in ways that greatly enhance their productivity and the quality of their code. There are things that composer should definitely NOT be used for (e.g. enabling extensions, expressing extension dependencies on other extensions, or in most cases autoloading code), and that current extensions should be updated in light of, but there are aspects of composer that are quite beneficial.

Since there has not been any similar functionality added to MediaWiki itself (nor should there be if an existing open source solution exists to provide that functionality), it seems reasonable that, rather than banning the use of composer, there be an accepted, well-documented approach to how extensions can use composer. Composer support should be optional, and it is fine if there is a limitation on extensions loaded in the Wikimedia infrastructure that prevents extension installation with composer.

Third party developers that I have spoken to are willing to make changes in their use of composer to comply with a reasonable policy when accepted, but would be dramatically against any attempt to remove all support for composer to install extensions.

daniel added a comment.EditedMay 27 2020, 9:04 PM

We could make the next Wednesday's TechCom office hour about this RFC. Traditionally, the first meeting each month is at a "Europe friendly" time, so this would would be at [EDITED to correct time] 16:00 UTC (9am PDT, 18:00 CEST). Does this work for everyone?

Sounds good. I won't be able to make it at that time, but if enough other people can I'll look forward to reading the logs.

@daniel Do you mean 16 UTC or is the plan to swap planning and discussion?

@daniel Do you mean 16 UTC or is the plan to swap planning and discussion?

Gah. You are right, sorry. I'll edit.

Krinkle updated the task description. (Show Details)Jun 3 2020, 4:09 PM
Foxtrott added a comment.EditedJun 3 2020, 4:18 PM

@Krinkle, If you want to change the description like that, I think you should include a definition of what exactly you mean by "installation". Otherwise this might come off as an attempt to obfuscate the obvious drawbacks of using only extension.json without composer.json. I think most people would understand downloading an extension and putting the wfLoadExtension into LocalSettings as "installation".

Edit: ... or in fact the obvious drawbacks of this hybrid solution

As I understand it, this proposal would introduce a third download mechanism for extensions. (after Git, and tarballs/ExtensionDistributor).

Comparing how they would work:

RequirementsInitial installUpdateSwitch version
TarballsDownload from mediawiki.org GUI and unpack where neededReplace directoryReplace directory
GitCLI access, Gitgit clone ext-url ext-versiongit pullgit checkout
ComposerCLI access, Composer + Gitedit composer-local.json and run composer updatecomposer updateedit composer-local.json and run composer update

In terms of installilng / managing extensions with MW. They all require the same: Edit LocalSettings.php and add wfLoadExtension() for each extension. Correct?

In terms of relationships between extensions, the RFC says Compooser will not be used to track dependencies between extensions. This would mean dependency-extensions would also need to explicitly mentioned in composer-local.json. Correct?

From the task description:

Advantages:

  • Centralized vendor/ directory: Extensions can store their dependencies in a centralized directory, rather than having them in their respective sub-directory.

This is already the case today. The exception is if someone downloaded extensions using the tarball method (which bundles vendor locally, this RFC doesn't change that, and that's also really hard to avoid in general).

If you run composer locally (docs) then afaik we only have one documented best practice, and that is to run composer update from the central directory (with wildcard from composer-local). Not doing so is afaik unsupported and would cause all sorts of issues (in addition to being a lot of work), because it means Composer doesn't have a full picture ands thus can't resolve dependency ranges very well, race conditions of who wins the autoload include etc.

Tgr added a comment.Jun 3 2020, 5:56 PM

Composer CLI access, Composer + Git

Git is not strictly needed, composer-local.json determines everything else (well, other than MediaWiki itself). It's a single file, so all kind of naive versioning strategies could work reasonably well, or someone could write a CLI or web tool for generating it (so you can click together what extensions you want on a rich web interface with descriptions, recommendations etc).

CLI access is also not strictly needed, there's stuff like Composercat (whether that's a good idea is another question).

I guess the main weakness is that you'll still need either tarballs or git for installing MediaWiki itself. It would be great to make MediaWiki installable via Composer, but that's a wholly different level of effort, and this RfC has wisely chosen not to concern itself with that.

Composer CLI access, Composer + Git

Git is not strictly needed, composer-local.json determines everything else […]

Users don't need to interact with Git commands when using Composer, however Composer itself requires Git afaik to be able to perform all its expected functions (e.g. not only tagged releases, but also dev-master, dev-REL1_x). In particular because WMF-developed extensions do not tag releases for anything other than core.

cscott added a subscriber: cscott.Jun 3 2020, 10:00 PM

Users don't need to interact with Git commands when using Composer, however Composer itself requires Git afaik to be able to perform all its expected functions (e.g. not only tagged releases, but also dev-master, dev-REL1_x). In particular because WMF-developed extensions do not tag releases for anything other than core.

I don't think this is strictly the case -- from https://getcomposer.org/doc/05-repositories.md#git-alternatives it appears that composer can download github-hosted files as ZIP tarballs using the github API. Since WMF-developed extensions are mirrored on github, that means that (in theory, I haven't tested this) you ought to be able to pull zips corresponding to release branches of WMF extensions without having git installed on your host. In any case, I think git ought to be considered a dependency of composer, not of this download mechanism: there are plenty of other ways you might be able to use composer to download extensions independent of git. We shouldn't confuse "how WMF would use composer to download WMF extensions (if WMF were to do so)" with "how third-parties might use composer to download third-party extensions"; the latter certainly doesn't *require* git.

I'd also like to recommend (mirroring what @CCicalese_WMF said in the RFC meeting) that we avoid using the term "install" in the description of this RFC, as potentially confusing w/ multiple meanings. "Download", "unpack", and "enable" seem to cover all the steps involved and are less ambiguous.

Akuckartz added a subscriber: Akuckartz.
bd808 added a subscriber: bd808.Jun 15 2020, 8:37 PM

There is an implicit software dependency for this RFC that I worry most folks have not thought about or understood. The composer-merge-plugin software which is currently required by mediawiki/core.git's composer.json only exists to make Composer management of extensions and skins possible. This software was written by me (@bd808) in the Foundation's FY2014/2015 so that we could start using composer.json to manage PHP library dependencies for MediaWiki core without destroying the possibility of using Composer to manage other software in a given MediaWiki deployment. Composer-merge-plugin is functionally abandonware today. No Wikimedia Foundation team is actively claiming "ownership" of the project. If this RFC were declined and T249573: RFC: Remove ability to install extensions and skins with Composer were accepted, we could remove the dependency on composer-merge-plugin from mediawiki/core.git.

If this RFC is accepted, in my opinion, there must be a Wikimedia Foundation team or long term stable MediaWiki technical community group established to be the "owner" of composer-merge-plugin. This is especially topical at the moment as Composer is preparing for an API breaking 2.0 release upstream and composer-merge-plugin needs active coding and testing to ensure compatibility with those upstream changes.

Unless I'm missing something, composer-merge-plugin is currently the only way cleanly and sanely set up extensions in a development environment. @bd808 is there an alternative to merge-plugin for that use case? Running composer for each extension only works if there are no version conflicts...

bd808 added a comment.Jun 17 2020, 9:36 PM

Unless I'm missing something, composer-merge-plugin is currently the only way cleanly and sanely set up extensions in a development environment. @bd808 is there an alternative to merge-plugin for that use case? Running composer for each extension only works if there are no version conflicts...

It only works via composer-merge-plugin if there are no version conflicts as well. I'm not saying that the tool is not useful, I'm saying that it is abandonware today and this RFC is codifying the need for it to exist perpetually.

It only works via composer-merge-plugin if there are no version conflicts as well. I'm not saying that the tool is not useful, I'm saying that it is abandonware today and this RFC is codifying the need for it to exist perpetually.

Unless I'm misunderstanding the RFC, I believe it is saying that the plugin would no longer be necessary as composer would be used to download the extension itself (and all it's dependencies).

No, the plugin would still be necessary, as people would need to be able to add extensions to their composer.local.json.

That said, even if this RFC is declined, the Composer Merge plugin is still required because there still needs to be a way to load extensions' dependencies. Because of that, I don't see that the lack of maintenance for the merge plugin is necessarily a stopper for this RFC (although I totally agree that it'd be good if there was some clarity around who maintains it).

I guess an other way to do it would be to remove composer.json from core (move it to composer.dist.json or something) and let wiki admins make their own composer.json. No merge plugin required, but other things would have to change.

Unless I'm misunderstanding the RFC, I believe it is saying that the plugin would no longer be necessary as composer would be used to download the extension itself (and all it's dependencies).

Without composer-merge-plugin, a local diff to mediawiki/core.git's composer.json file would be required to composer install an extenstion. That untracked state will be lost if a new tarball is unpacked over the existing state in the case of a tarball based deploy. With a git based deploy such state could be tracked as a local commit that is rebased on top a a fresh pull, but that workflow will also be awkward.

Use of composer require ... to automate both the composer.json edit and the install in one action is not covered by this RFC currently and would not be corrected by the presence of composer-merge-plugin. That could be seen as a deficiency in the current discussion as it not addressed at all. I will leave it to the group to decide if the desired outcome of this RFC is actually a fully documented and tested workflow or simply retention of the current functionality that was proposed for removal in T249573: RFC: Remove ability to install extensions and skins with Composer.

@dbarratt, your own proposal in T166956: Cannot use Composer's CLI to manage a project's dependencies would be needed to cleanly allow both outcomes without helper software such as composer-merge-plugin.

@bd808 I see. apologies for my confusion.

If this RFC is accepted, in my opinion, there must be a Wikimedia Foundation team or long term stable MediaWiki technical community group established to be the "owner" of composer-merge-plugin.

Is there a phab task or discussion somewhere that is tracking that possibility? It's a fairly widely used plugin so if we are not going to keep using it, we should probably try to find a new maintainer.

Tgr added a comment.Jun 20 2020, 10:11 PM

Composer-based extension management would actually be fairly easy without composer-merge-plugin:

  • move MediaWiki's composer.json out of the way (rename to composer.dist.json, for example)
  • create a Packagist module for MediaWiki core which contains nothing other than core's composer.json file
  • people who install MediaWiki via Composer would have to add that module, alongside their extensions and skins.

(Which is basically a lightweight / hacky version of T166956: Cannot use Composer's CLI to manage a project's dependencies.)

But it's a moot point, because we need to keep composer-merge-plugin alive for git-based installs anyway.

Tgr added a comment.Jun 20 2020, 10:13 PM

This is especially topical at the moment as Composer is preparing for an API breaking 2.0 release upstream and composer-merge-plugin needs active coding and testing to ensure compatibility with those upstream changes.

The task for that is T248908: Test composer-merge-plugin against Composer 2.0 pre-release, FWIW.

I think moving forward with composer and endorsing it as a supported way to distribute and install extensions is a mistake. It feels like we're taking this half-baked idea that just keeps dropping individual components (can't use composer.json in core, moved to composer-merge-plugin/composer.local.json; can't load entrypoints, now dropping that). I think composer is an adequate development tool, but I don't think it meets our needs for extensions. I also think this process is generally backwards: we should be clearly documenting what our needs and wants are (T118188), and then evaluating different solutions against that, rather than picking composer as the solution and retroactively building requirements around that.

I break down extension management into 3 areas: discovery, installing, upgrading.

Discovery:

Pros:

  • packagist is something that already exists, that we can use
  • packagist supports any Git repo that is accessible to the Internet

Cons:

  • Packagist/composer are English-only, and don't support any localization.
  • Packagist is filled with plenty of other PHP stuff, so you need to first filter against MediaWiki things before looking for what you want.
  • Packagist is hosted by a 3rd party with an unfavorable privacy policy (uses Google Analytics)

Installing:

Pros:

  • MediaWiki developers and sysadmins who use composer are already familiar with it.
  • There's plenty of documentation out there for people who have trouble with composer
  • Composer can automatically resolve library dependencies, if they are compatible.

Cons:

  • Composer is command-line only, leaving behind people who use the web installer or aren't that comfortable with CLI
  • Composer introduces multiple third-parties into the integrity verification pipeline, compared to downloading from Wikimedia Gerrit/tarball
  • It's really easy to use the wrong way, regardless of how experienced you are (see: --no-dev and the PHPUnit RCE affecting prominent wikis run by very experienced MW developers)

Upgrading:

Pros:

  • If everything is compatible, it's able to update everything at once, with little interaction needed

Cons:

  • ..? (I'm sure there are some, I can't think any right now)

Based on the way I read the proposal, the proposed usage of composer is down to:

  • a registry
  • automated install tool based on list of names
  • automated updating based on list of names
  • automated fetching and set up of composer dependencies

Am I missing anything? At that point, I think we're better off writing our own dependency manager thing (as I suggested/prototyped before). Even one of the multi-git wrappers would probably be preferable. IMO composer's strength is in it's dependency resolver and semver, but most MediaWiki extension dependency trees are at most one level deep, and rarely conflict with each other (since we expect master always works). Most MediaWiki extensions don't have composer dependencies, so I think optimizing for that case is wrong.

I think (hope?) all of us still dream of the day when extension management is as simple as it is in WordPress, but I fear that going down this composer road is just going to make that dream even more of a pain to achieve.

Legoktm added a comment.EditedJun 24 2020, 7:59 AM

Technical problems with composer extensions:

  • We currently autoload composer late, for the purpose of supporting extensions, which causes problems with using library code in MediaWiki itself (see https://lists.wikimedia.org/pipermail/wikitech-l/2019-June/092233.html, which has been worked around for now)
  • Having an extension dependency in composer.json messes up the normal CI flow of composer install && composer test because there's now an unexpected ./extensions/Foo directory. This is ususally worked around by teaching each tool to ignore the non-standard extensions directory.

These are off the top of my head, there are definitely more that we've worked around over the years. None are unfixable, but I don't think this is as straightforward as "composer works great, let's codify it".

Surely there are better options, but they have yet to be implemented (cf. @Tgr's comment above). So composer seems to be a viable way to go, it has a solid usage basis and a set of MediaWiki developers willing to maintain it.

See T118188#2613011 / https://gerrit.wikimedia.org/r/c/mediawiki/core/+/308891 - it's hard to move forward when you're met with silence...

Most MediaWiki extensions don't have composer dependencies, so I think optimizing for that case is wrong.

Many of them do, and I think even more would do, if it weren't that cumbersome to add (required non-dev) composer dependencies to WMF deployed extensions.

Regardless whether we go with another solution or composer, I think the solution should include way of installing composer deps and alert about incompatible requirements (ideally across core, extensions and their composer deps).

Most MediaWiki extensions don't have composer dependencies, so I think optimizing for that case is wrong.

Many of them do, and I think even more would do, if it weren't that cumbersome to add (required non-dev) composer dependencies to WMF deployed extensions.

My hastily written script (P11646) found 7% (59/833) of extensions in Wikimedia Gerrit have composer dependencies - I think that's pretty minimal. I do agree that it's likely some more would have composer dependencies if it were straightforward to do so,

Regardless whether we go with another solution or composer, I think the solution should include way of installing composer deps and alert about incompatible requirements (ideally across core, extensions and their composer deps).

Agreed.

Surely there are better options, but they have yet to be implemented (cf. @Tgr's comment above). So composer seems to be a viable way to go, it has a solid usage basis and a set of MediaWiki developers willing to maintain it.

See T118188#2613011 / https://gerrit.wikimedia.org/r/c/mediawiki/core/+/308891 - it's hard to move forward when you're met with silence...

Putting my money where my mouth is, here's another prototype: P11658 (Python because I'm faster in it than PHP still, but it should be trivially translatable). It supports git (using REL1_XX branches) and extdist and having composer dependencies. We could probably figure out a scheme for semver git tags too. It figures out what the locally intalled version is, what the latest version available for download is, and updates if necessary. If there are composer dependencies, it'll trigger composer update. In theory if it went wrong we'd be able to rollback as well. Here's an example of what it looks like if you're moving from REL1_31 to REL1_34.

$ ./mwext.py update
Getting the latest version of TemplateStyles
/home/user/projects/mwext-prototype/test/extensions/TemplateStyles needs an update
Getting the latest version of TemplateStyles
[extdist]: Updating /home/user/projects/mwext-prototype/test/extensions/TemplateStyles...
[extdist] Downloading https://extdist.wmflabs.org/dist/extensions/TemplateStyles-REL1_34-c4d6f25.tar.gz
[extdist] Successfully updated /home/user/projects/mwext-prototype/test/extensions/TemplateStyles
Getting the latest version of AbuseFilter
/home/user/projects/mwext-prototype/test/extensions/AbuseFilter needs an update
Getting the latest version of AbuseFilter
[git]: Updating /home/user/projects/mwext-prototype/test/extensions/AbuseFilter...
Previous HEAD position was 7b66a2c8 SECURITY: Unbreak blocks shorter than one hour
HEAD is now at 3c2035dd SECURITY: Unbreak blocks shorter than one hour
[git] Successfully updated /home/user/projects/mwext-prototype/test/extensions/AbuseFilter
Triggering composer update...
Loading composer repositories with package information
Updating dependencies
Package wikimedia/password-blacklist is abandoned, you should avoid using it. Use wikimedia/common-passwords instead.
Generating optimized autoload files

TL;DR: Perfect is the enemy of Good Enough.

Developing a custom extension manager will require having the WMF commit to maintaining it in the future. The approach proposed in this RFC would require continued maintenance of composer-merge-plugin - a component that, as noted in T250406#6233269, is also necessary for other reasons. Given how difficult it has been to find someone at the WMF to maintain that code, I am skeptical of the long-term maintenance of some yet-to-be developed code.

But, let's back up a second.

The goal of this RFC is simply to codify the best practices and anti-patterns of using composer to download extensions--something developers are already doing. This has been an open area of concern for many years, and an alternative approach to composer has not materialized.

Perhaps in the future, such an approach will indeed be developed, but in the meantime, documentation is needed for the best approach in the current situation.

If another solution with WMF maintenance buy-in appears, that's great, but we (third party users like SMW devs and Hallo Welt!) need to codify what already exists.

TL;DR: Perfect is the enemy of Good Enough.

Developing a custom extension manager will require having the WMF commit to maintaining it in the future. The approach proposed in this RFC would require continued maintenance of composer-merge-plugin - a component that, as noted in T250406#6233269, is also necessary for other reasons. Given how difficult it has been to find someone at the WMF to maintain that code, I am skeptical of the long-term maintenance of some yet-to-be developed code.

A properly developed extension manager would actually be used by Wikimedia production, therefore getting the maintenance and love it needs. Adopting composer would suffer from the same problem you explain - Wikimedia production isn't going to adopt it. That's fundamentally the same state composer-merge-plugin is in, it's used for a specific featureset that CI uses, and that's just about what's maintained.

But, let's back up a second.

The goal of this RFC is simply to codify the best practices and anti-patterns of using composer to download extensions--something developers are already doing.

I fundamentally disagree with using composer to manage extensions as an officially supported method by MediaWiki. There are significant unaddressed flaws with this method (partial list in my previous comments) that are being ignored because the people using them don't perceive them as problems, even though they are or have gotten frustrated enough to work around them.

This has been an open area of concern for many years, and an alternative approach to composer has not materialized.

Perhaps in the future, such an approach will indeed be developed, but in the meantime, documentation is needed for the best approach in the current situation.

If another solution with WMF maintenance buy-in appears, that's great, but we (third party users like SMW devs and Hallo Welt!) need to codify what already exists.

I would posit that the constant pushing of a flawed composer-based solution has taken any remaining energy from potential alternatives, but I don't want to get into that - moving forward is more important.

In any case, if the goal is just documentation...just document it. https://www.mediawiki.org/wiki/Composer/For_extensions exists, so update it. As long as people know what they're getting into (the warning banner at the top of the page), extension maintainers can distribute their extension however they want. It just won't be officially supported by MediaWiki, with the potential for breaking changes caused by subtle initialization changes, etc.

Osnard added a comment.Tue, Aug 4, 6:59 PM
  • We currently autoload composer late, for the purpose of supporting extensions, which causes problems with using library code in MediaWiki itself [...]
  • Having an extension dependency in composer.json messes up the normal CI flow of composer install && composer test because there's now an unexpected ./extensions/Foo directory. [...]

This RFC wants composer to be used for installation and update of MediaWiki extensions and their respective libraries. By this RFC an extension must not have any "static" files be included by the autoloader. Not to enable itself, not to include any default settings or manipulate/access the global state. Not even the PSR-4 autoloader of composer should be used. That's what the ExtensionRegistry is for. The RFC states that "inter-extension-dependencies" are only to be declared through ExtensionRegistry. If the "requirements" of this RFC get implemented, the issues mentioned above should be prevented. Simple CI tests could be introduced that make sure those rules are being followed.

The important thing is, that these practices get to be known and followed by extension developers. Therefore they should be "official". Third-party developers look at what the WMF recommends and what the WMF actually does in their extensions.
The only good way to encourage developers to use libraries is to also allow them to install the extensions using composer. Thus having library dependencies and constraints properly checked and the code installed in a central location.

I just want to get over "you must not use composer for anything else than CI tests" and come to a commonly accepted "using composer for installation is okay if you follow these simple rules" situation. But this requires a little support by the WMF. And that's what this RFC is about.

I would like to see MediaWiki core, libraries, extensions and skins installable by composer -- just like you can do with Drupal (introduced in Drupal Core 8.8 and mentioned earlier in this RFC). "But wiki extensions don't use semantic versioning." Drupal modules (mostly) don't follow semantic versioning, but there's a shim to accomodate that. End users aren't forced to use composer - because it's not the only way to download/install/manage Drupal. In fact, it's not even mentioned on the project download page. The point is, I wholeheartedly support this RFC in terms of improving and clarifying composer support in the MediaWiki ecosystem.

As an extension developer, I want to be able to user composer for my extension dependencies, and I want those to be recognized by MediaWiki. At the present time, we need to declare dependencies in extension.json and composer.json, (and in zull/parameter_functions.py for CI)

The warning at the top of https://www.mediawiki.org/wiki/Composer/For_extensions seems heavy-handed. The declined RFC was from 2013! The emphasis on "is likely to be removed" needs a citation.

I would like to see MediaWiki core, libraries, extensions and skins installable by composer -- just like you can do with Drupal (introduced in Drupal Core 8.8 and mentioned earlier in this RFC).

Also, if you "require" an extension with Composer in Drupal, it does not enable that extension. I think the same should happen for MediaWiki.

Also, if you "require" an extension with Composer in Drupal, it does not enable that extension. I think the same should happen for MediaWiki.

That is the current state: If an extension is required it is not enabled. We've worked with extension developers to ensure that it isn't activated automatically.