Change 551346 had a related patch set uploaded (by Jforrester; owner: MaxSem):
[mediawiki/core@master] Rm support for extensions requiring a MW version via Composer

https://gerrit.wikimedia.org/r/551346

@Reedy can you flesh this out a bit more, so it follows the guideline at https://www.mediawiki.org/wiki/Requests_for_comment, and move it to the appropriate phase (probably phase 2).

The situation with respect to use of composer in extensions has changed significantly since T467. I used to be opposed to composer use in extensions myself, but I have been convinced by seeing how composer is currently used in extensions. Today, the use of composer by some third parties to install extensions and maintain their infrastructure is quite impressive.

One of the early objections to composer use had to do with bypassing extension registration and using composer to enable extensions, but even Semantic MediaWiki has changed so that it no longer automatically enables itself. It is important to differentiate between installing an extension, which can supported in many ways, and enabling an extension.

There are many third party MediaWiki extension developers who are using composer in ways that greatly enhance their productivity and the quality of their code. There are things that composer should definitely NOT be used for (e.g. enabling extensions, expressing extension dependencies on other extensions, or in most cases autoloading code), and that current extensions should be updated in light of, but there are aspects of composer that are quite beneficial.

Since there has not been any similar functionality added to MediaWiki itself (nor should there be if an existing open source solution exists to provide that functionality), it seems reasonable that, rather than banning the use of composer, there be an accepted, well-documented approach to how extensions can use composer. Composer support should be optional, and it is fine if there is a limitation on extensions loaded in the Wikimedia infrastructure that prevents extension installation with composer.

Third party developers that I have spoken to are willing to make changes in their use of composer to comply with a reasonable policy when accepted, but would be dramatically against any attempt to remove all support for composer to install extensions.

I agree with everything @CCicalese_WMF has said above. Let's figure out a middle ground here; installing extensions with Composer is really useful!

Certainly, get rid of having an extension be able to require a version of MediaWiki; that's not too widely used and creates a slightly strange looping of dependencies.

But is this also proposing to remove the composer-merge plugin? In which case, how will extensions require other packages? Will each extension have their own vendor directory? What about clashing versions between extensions?

In T249573#6064277, @Samwilson wrote:

But is this also proposing to remove the composer-merge plugin? In which case, how will extensions require other packages? Will each extension have their own vendor directory? What about clashing versions between extensions?

This is definitely the supported part (requiring non MW skins/extensions 3rd party PHP libraries) of using composer in MediaWiki.

Even if we consider composer-merge-plugin a hack, I think there other tasks about finding better ways to handle this including/munging of sub dependancies into vendor

Oh right, that's good to know. Then it sounds like we'd have to remove the mergability of the require section of composer.local.json if we wanted to prevent people from installing extensions with Composer. Extensions would still have their own require section of course, but only include would work at the higher level.

In T249573#6063528, @CCicalese_WMF wrote:

[…] It is important to differentiate between installing an extension, which can supported in many ways, and enabling an extension.

[…] There are things that composer should definitely NOT be used for (e.g. enabling extensions, expressing extension dependencies on other extensions, or in most cases autoloading code), and that current extensions should be updated in light of, […]

I think it's fine for wikis to continue to use Composer to download the source code of extensions hosted on Packagist. MediaWiki is quite flexible around where the code comes from or where it lives on disk.

Would the removal of the current partial support affect that use? I assume it does, but it's not clear to me how. More detail on the specific problems that come up would help :)

Would the removal of the current partial support affect that use?

This is how the conversation floundered last time, I think! Because no, removing support for "specifying MediaWiki as a requirement of an extension" does not stop extensions from being installed with Composer. The patch above that MaxSem started is probably a good thing and should proceed, but not under the title of "remove ability to install extensions", because it doesn't do that. :)

If this is to be an RfC, it needs proper rationale and cost-benefit analysis, not just a "hey let's break this functionality that an unknown fraction of MediaWiki users rely on". It's hard to even tell from the description what functionality is being broken.

Based on the patch, that functionality is MediaWiki exporting itself as a package (ie. pretending it was installed with something like composer install mediawiki/mediawiki 1.35.0, despite not actually being installable that way, for the purpose of allowing extensions to specify the supported version). So 1) all extensions specifying MediaWiki as a requirement in their composer.json that way would have to be updated (quick usage check: 3 in codesearch; I can find more on GitHub, but it's search syntax is not useful for getting a comprehensive result set), 2) after that, those extensions would still be installable via composer the same way they were before (so not sure if this would affect expectations at all), 3) the installation process would be more fragile (as composer wouldn't tell you you are installing an imcompatible extension; the extension registry has a similar check, but that's only invoked when running update.php, which is not a requirement for simple extensions, or when using the site, which is a bit too late). I don't see any obvious benefit there (unless the removed code was a maintenance burden).

That said, maybe a real composer package would be a better place for this code? People wishing to use MediaWiki via Composer could just composer install mediawiki/mediawiki (or mediawiki/mediawiki-composer-compat or whatever) which would not actually install MediaWiki but ensure Composer-based extensions work with it, while decoupling maintainer responsibilities for the Composer-specific logic and MediaWiki itself.

I ran into this when cleaning up includes/ (T321882). You can bring back composer supported functionality via a new repo that uses core as submodule or anything like that. So I'm planning to merge this patch soon.

In T249573#8759975, @Ladsgroup wrote:

I ran into this when cleaning up includes/ (T321882). You can bring back composer supported functionality via a new repo that uses core as submodule or anything like that. So I'm planning to merge this patch soon.

This whole discussion is a bit hard to follow. Could you summarize what this means explicitly for extension writers (if anything)? Will this break things that people are actually doing in practice? Do people who maintain extensions have to take any action as a result of this change?

[For the avoidance of doubt, I'm not objecting, I just want it explicitly documented what this change means for extension writers]

Since either 1.22 or 1.23, there has been a composer hook that allowed extensions to depend on mediawiki/mediawiki in their composer.json, as a way of specifying what MW version they need (aka what extension.json's "requires" does). Codesearch only finds 2 uses left: https://codesearch.wmcloud.org/search/?q=mediawiki%2Fmediawiki%22&files=composer.json&excludeFiles=&repos=

Once the hook is removed, composer installation of those extensions will fail because mediawiki/mediawiki is not a real package on packagist. Presumably the fix is to remove those requires entries, adding it to extension.json if necessary, and releasing a new version.

I think this task should be renamed to RFC: Remove ability to load extensions and skins with Composer because it isn't really about installing as far as I can tell (although I guess installing = downloading + loading — it's the downloading part that isn't relevant here).

The above patch will mean that extensions will not be able to declare that MW core is a dependency (which is a great idea). It won't stop extensions from loading themselves when installed, will it? I'm not sure it's very easy to stop that: an extension just has to put wfLoadExtension() in a file that's autoloaded in composer.json (e.g.). But that does seem to be rare these days (e.g. Maps stopped doing that in 2018).

In T249573#8761222, @Samwilson wrote:

I think this task should be renamed to RFC: Remove ability to load extensions and skins with Composer because it isn't really about installing as far as I can tell (although I guess installing = downloading + loading — it's the downloading part that isn't relevant here).

No, removing that was agreed on in T467 (and most extensions which used to support autoloading via Composer have stopped doing that after that RfC). This RfC is specifically about installing (downloading, if you like).

The above patch will mean that extensions will not be able to declare that MW core is a dependency (which is a great idea).

It will mean that specifying supported core versions cannot be done in composer.json, and consequently composer cannot automatically select of the latest extension version that's compatible with installed versions of core, other extensions, PHP etc.

This RfC is specifically about installing (downloading, if you like).

Oh right, sorry I've misunderstood.

composer cannot automatically select of the latest extension version that's compatible with installed versions of core, other extensions, PHP etc.

That's true, but it still doesn't stop people installing extensions with Composer. They might end up with incompatible ones (and avoiding that is a matter of appropriate version numbers for extensions, I think, i.e. bumping the major number when dropping support for older versions of MediaWiki… but that's a whole other topic).

With the above patch applied to core, adding the following to composer.local.json still works to download the extension to extensions/ApprovedRevs:

"require": {
	"mediawiki/approved-revs": "^1.8"
}

(Subscribers of this task might also be interested in T333668: Shared tools for common development environment tasks which proposes to add this type of functionality via maintenance scripts in core.)

So, I sort of had a change of heart about this task. (To be clear because it is confusing, by this task, i mean removing support for depending on the fake mediawiki/mediawiki dependency in composer)

I do not like the trend of using composer to manage extensions. However, it seems like that trend is here to stay. Since people are going to do it anyways, we should make it not suck.

Maintaining compatibility with different versions of MediaWiki is hard. That's why we normally use REL1_XX branches. If people are installing with composer, one of the purported benefits is composer can select the correct version for you. In SemVer, the idea is if you update your dependencies (which mediawiki is from the perspective of an extension) but otherwise don't change your program, that is a patch release - users can then use the new version if the dependencies work better for them, with confidence that the program itself hasn't had breaking changes. Composer can just chose the right version.

That is entirely broken if you cannot specify mediawiki dependency in your composer.json.

So if we remove this feature, it kind of feels like we're keeping composer around for this usecase, just making it be terrible.

I assume the current lack of use of this feature is largely around the FUD that it might not be stable, which might change if we officially bless it.

edit: I misunderstood how this feature worked. I take all of this back.

I don't know how technically feasible it is but I think it should be possible but out of core, into a dedicated repo that could have core as a submodule. That way it removes the maintenance burden on people who are maintaining core for Wikimedia but at the same time, allows people to maintain the composer repo for third party usage. Basically moving overhead of third parties to people who are willing to maintain it. Don't know how feasible it is.

I don't know how technically feasible it is but I think it should be possible but out of core, into a dedicated repo that could have core as a submodule.

That doesn't sound that practical to me for this usecase, but perhaps i lack imagination.

Lets take a step back here, what is the problem we are trying to solve?

As far as i can tell, there are a couple separate issues that are being inter-tangled here.

A) Does it make sense to have this fake pseudo-mediawiki composer dependency at all.

B) Does the complexity induced by support for this feature outweigh its benefits

C) Does the maintenance burden of this feature outweigh its benefits

D) Does the maintenance burden on wikimedia foundation outweigh its benefits to the wikimedia foundation.

A) seems like something nobody has really talked about. I want to call it out because i think its important to separate "is this code a good idea in principle" from "The code is too costly" For b), the complexity seems minimal as it is very isolated from all other code, and the code is not complex.

It seems like we are mostly explicitly talking about c), but the actual concern is d).

From looking at this code, it seems very low maintenance in the past, and that seems unlikely to change in the future. Proposals like switching the code to run in a submodule somehow, don't really reduce maintenance burden (They would probably increase it), just transfer maintenance burden to other people. In terms of any "maintenance crisis", removing this component seems like a micro-optimization unlikely to solve anything.

So i think the real concern is d). In essence, in current mediawiki, wikimedia folks serve as a backstop for all maintenance of everything, and this is arguably unfair in what should be a shared burden. I think this is a social problem, and we need to look for social solutions first, before looking at technical solutions.

What i would generally propose, is we should have directory level (or even file level as the case may be) "maintainers", who are expected to do maintenance as it comes up for the things they own (based on bugs filed). If that doesn't happen it becomes un-owned, and if bugs/maintenance request pile up, someone can start a process like code stewardship, there is some time period where a new owner can volunteer to take over, or if nobody volunteers the code gets removed.

There's been attempts at this in the past, the mw.org Maintainers page, but they've been mostly BS, with unclear responsibilities and hence have been pretty much useless.

Anyways, i think this would be a fair way to make responsibility shared, without it feeling unilateral. I also think its the only way to prevent all the burden from falling on a small group of people. When everybody is responsible, nobody is responsible - lets change that.

Sorry, i just realized i misunderstood how this feature worked, and I don't think it is a viable solution to the problem it is trying to solve. I still stand by my comment about code-owners being a path that we as a community should move forward on. However i no longer object to this feature being removed, as i don't think it is a viable way forward to the versioning problem, since using it means you cannot install the extension independently of composer, which is kind of critical to how the majority of people use composer. So consider my objection withdrawn.

Edit to clarify: I mean, if you depend on mediawiki/mediawiki, that means it is now impossible to run tests with composer because you cannot install dev dependencies. It also prevents the strategy of not using composer merge plugin, and having extension.json load the composer autoloader. I think these are critical use cases than any solution would have to respect, and this one doesn't, so I'm not on board with it anymore.

The component had 13 changes in the last five years, for a total of 120 lines. That's a pretty modest maintenance burden (the discussion here probably took more time :). The majority of it was non-specific update work (e.g. changing function signatures) which isn't really feasible to split up between maintainers.

OTOH when the discussion started, there were only 3 extensions (known to codesearch) using this pattern and now there is only one, so it does seem like people are voting with their feet.

In T249573#8830446, @Bawolff wrote:

It also prevents the strategy of not using composer merge plugin, and having extension.json load the composer autoloader.

FWIW that's a pretty bad strategy we should move away from anyway. If two extensions use the same library via composer and require different but overlapping version ranges, merge-plugin can handle that, but with separate composer runs you end up with ugly and hard-to-debug errors as one of the extensions might end up with an unexpected library version. The solution would be something akin to T167038: Separate "application" and "project" concerns but that's way more effort than the 3rd-party community has historically been able to muster.

I think a simpler solution to the problem of Composer installing compatible extensions is for extensions to increment their major version number when they drop support for a version of core. This means that the site owner has to actually update the number in their composer.local.json and so will know to go and read release notes when they do. In addition, extensions can specify a minimum and maximum supported version of core (which is not something that very many do, I think, but it's definitely supported). That means that even if an extension gets installed within an unsupported version of core, the failure is immediately obvious and the wiki doesn't half-function.

Pretty much the same applies for compatibility with other extensions.

(Of course, relying on version numbers doesn't help that much when they're not updated for years on end or even not tagged at all.)

I think a simpler solution to the problem of Composer installing compatible extensions is for extensions to increment their major version number when they drop support for a version of core.

I feel like this defeats the benefit of SemVer. Users can no longer tell if your update is major or not.

I mean, if you depend on mediawiki/mediawiki, that means it is now impossible to run tests with composer because you cannot install dev dependencies. It also prevents the strategy of not using composer merge plugin, and having extension.json load the composer autoloader.

Oh, i guess the solution to this could be - instead of saying your extension requires a specific version of mediawiki/mediawiki, say it conflicts with versions you do not support. This should still allow installing independently but should still allow composer to choose right version.

Too bad composer doesn't support npm's idea of peer dependencies.