In T88596#1018332, @mwjames wrote:

I agree that this is a real issue. Since you have experienced this pain first hand, would you be willing to contribute some tests?

I got thrown under the bus by some MW developers arguing that I should not try to reason "over tests that they themselves have not produced"

Hm. did this discussion take place in a public place like a bug report or a mailing list? I had no idea about this problem, and it would be useful to understand what happened in order to avoid a similar type of frustration in the future.

We had an IRC meeting about this on Wednesday, February 18. Here are the MeetBot minutes:

wikimedia-office: RFC meeting

Meeting started by TimStarling at 21:00:37 UTC. The full logs are
available at
https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-02-18-21.00.log.html
.

Meeting summary

• '''Improving extension management | RFC meeting | Wikimedia meetings channel | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/''' (TimStarling, 21:01:02)
  • ''LINK:'' https://www.mediawiki.org/wiki/RFC_metadata (TimStarling, 21:01:15)
  • ''LINK:'' https://www.mediawiki.org/wiki/Requests_for_comment/Improving_extension_management (legoktm, 21:03:30)
  • adding compatability data to extension.json: probably fine (TimStarling, 21:33:17)
  • general consensus that install via cli with good instructions it a good place to start (bd808, 21:33:29)
  • on having our own extension update system which uses composer classes -- I think we have consensus (TimStarling, 21:34:17)
  • use composer for lib dependencies. use bryan's merge-dependencies thingy or something similar. (DanielK_WMDE_, 21:35:16)
  • on dependencies between extensions -- I don't think there is consensus on whether to specify those dependencies in extension.json or composer.json (TimStarling, 21:36:55)

We had an IRC meeting about this on May 13. Here is a meeting summary written by Matthew Flaschen which was posted to wikitech-l:

The focus of the meeting was two aspects:

1. A way for the extension to specify which version(s) of MediaWiki core it worked with.

This was the focus of the majority of meeting. Basically, this would let an extension specify that a given commit (and thus, a given branch) worked with particular versions of core, using syntax like:

"supports": ">1.23<1.26"

Any syntax that Composer supports will work on the right.

There was a lot of support with this, with some desire for better communication, socialization, and documentation, and some debate over the key name. However, this was not unanimous. Outcome of this was "just submit a patch for it and we can continue the discussion in gerrit".

2. Tardist. Details at https://www.mediawiki.org/wiki/Extension:ExtensionDistributor/tardist . There wasn't much discussion of this, but some support based on having read it.

There was also some big-picture debate on whether we should be implementing a packaging/dependency system and various concerns:

1. Can we use Composer unmodified, whereas the current proposal/already implemented decisions is to use Composer for libraries, and build our extension system on top of it?

General reason we haven't done this is that extensions are not libraries, and have MediaWiki-specific concerns (e.g update.php). However, there was some push-back in today's meeting about whether this distinction is valid.

2. Should we instead use (or auto-generate) something like Debian packages, even if we don't actually try to get it into Debian proper. This means having our own Debian repo. People pointed out that to even start getting serious about this we have to at least package latest MediaWiki and provide it in a public repo. We haven't done this so far, and the latest MW in even Debian unstable is 1.19.20+dfsg-2.3 (latest release is 1.24, 1.25 coming out Real Soon Now).

Also, if Debian packages are our solution for manging extensions and core, what about people on Red Hat, Windows, BSD, etc.?

3. In general, has enough research been done on the existing packaging and dependency ecosystem to see if we can leverage an existing system?

Composer's json schema provides a way to extend it for special use cases
through the "extra" field:
https://getcomposer.org/doc/04-schema.md#extra

If there are extension use cases that are not compatible with Composer,
it would be good to make those explicit now so that people when people
ask "Why didn't you use composer?" in the future, we can point them to
this discussion.

Of course, if we have cases that composer doesn't fit, we may be able to
get the composer developers to a) show us how we're wrong or b) adapt
composer to fit our needs.

In T88596#1337745, @MarkAHershberger wrote:

If there are extension use cases that are not compatible with Composer,
it would be good to make those explicit now so that people when people
ask "Why didn't you use composer?" in the future, we can point them to
this discussion.

The biggest one for me is well discussed on a semi-related RfC specifically about using Composer for all extension management:

• https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Extension_management_with_Composer#A_way_to_disable_extensions_40627

Composer is great, perfect and awesome as a library management tool that allows freezing dependencies associated with a deployment. It is not however a general purpose plugin/extension management system.

In T88596#1337745, @MarkAHershberger wrote:

If there are extension use cases that are not compatible with Composer,
it would be good to make those explicit now so that people when people
ask "Why didn't you use composer?" in the future, we can point them to
this discussion.

This is covered in the RfC itself: https://www.mediawiki.org/wiki/Requests_for_comment/Improving_extension_management#Why_not_use_composer_directly.3F

I wrote a request for help for the one issue I am most familiar with in
Composer: https://github.com/composer/composer/issues/4109

I think this is the same issue as what @bd808 said.

Composer is great, perfect and awesome as a library management tool that allows freezing dependencies associated with a deployment. It is not however a general purpose plugin/extension management system.

It is certainly an opinion from the WMF side which as an external extension maintainer I do not support nor have I heard real arguments as to why an extension can't be managed by Composer alone. I have yet to see a real benefit from using something like extension.json in an environment where dependencies/versions do play an actual role beyond the point of maintaining a version number or static entry points for hooks.

mwjames writes:

It is certainly an opinion from the WMF side which as an external
extension maintainer I do not support nor have I heard real arguments
as to why an extension can't be managed by Composer alone.

Both @cicalese and I work on non-WMF MediaWiki projects. @cicalese runs
a wikifarm and I'm running single wiki install that will soon need to
host multiple wikis.

We've been discussing the composer-only solution
for wikifarms and
at least one developer there (no contradictory opinions have been voiced
from their side) seems to think that composer-only is the wrong solution
for extension management. If nothing changes, this records a non-WMF
reason for using something besides Composer.

I find this a bit disheartening since I thought Composer would be a good
base to build on for extension management.

@MarkAHershberger, can you be precise about what was the argument against using composer only in the discussion on github?

I just thought of an argument why using composer and another dependency manager at the same time might be a bad idea, at least if they are agnostic to each other.

As I understand, the plan is to have the MediaWiki extension manager install all extensions first and then invoke composer on each of them one after each other to install the libraries they need. In this case, if two extensions that need the same library, that library would be installed two times. What's worse, they may require it in different versions, but none of the two composer invocations would know of the constraints of the other extension. If composer was used to install extensions and their libraries, composer would know whether there is a version of the library that is compatible with both extensions and install it once, or complain if there is no such version.

Currently Mediawiki uses https://github.com/wikimedia/composer-merge-plugin to avoid what you described.

How does the plugin know which composer.jsons to take into account? Does the admin have to populate the include property in his composer.json manually or will the composer.local.json be generated somehow?

Which other composer.json files to merge is specified in composer.json recursively, with wildcards. In mediawiki/core that is composer.local.json , which then the user creates as e.g. merging extensions/*/composer.json and skins/*/composer.json or only specific ones.

Thanks for the explanation! Is this documented somewhere? Since the mechanism is already used in REL1_25, which is out, it would be good to have documentation for it...

In T88596#1785596, @Ingomueller-net wrote:

Thanks for the explanation! Is this documented somewhere? Since the mechanism is already used in REL1_25, which is out, it would be good to have documentation for it...

Documentation on wiki seems to be incomplete. There is a mention at https://www.mediawiki.org/wiki/MediaWiki_1.25#Better_support_for_composer-managed_extensions but no detailed explanation on https://www.mediawiki.org/wiki/Composer or https://www.mediawiki.org/wiki/Composer/For_extensions.

Some additional data may be gleaned from T67188: composer.json should be useable by WMF, core and extensions and rMW578f2cc8b0c1: Introduce Composer merge plugin.

I'll try to add some more description on the mw.o pages but would welcome the additions/corrections of others as always. T2001: [DO NOT USE] Documentation is out of date, incomplete (tracking) [superseded by #Documentation] is never done. :)

Ingomueller-net writes:

@MarkAHershberger, can you be precise about what was the argument
against using composer only in the discussion on github?

I previously said

at least one [composer] developer there (no contradictory opinions have been voiced
from their side) seems to think that composer-only is the wrong solution
for [MediaWiki] extension management.

As seen on the github issue, when I explained what wiki farms need from
composer -- enabling or disabling extensions based on the wiki being
used from a single code base -- the scenario was dismissed.

The alternative suggested was to have a different mediawiki installation
for every wiki.

With the developers of composer being unwilling to adapt it to
multi-site use cases like a wiki farm, I agree with the developers:
composer isn't suited for managing MediaWiki extensions.

In T88596#1785856, @bd808 wrote:

I'll try to add some more description on the mw.o pages but would welcome the additions/corrections of others as always. T2001: [DO NOT USE] Documentation is out of date, incomplete (tracking) [superseded by #Documentation] is never done. :)

I added https://www.mediawiki.org/wiki/Composer#Using_composer-merge-plugin

In T88596#1786284, @MarkAHershberger wrote:

As seen on the github issue, when I explained what wiki farms need from
composer -- enabling or disabling extensions based on the wiki being
used from a single code base -- the scenario was dismissed.

The alternative suggested was to have a different mediawiki installation
for every wiki.

With the developers of composer being unwilling to adapt it to
multi-site use cases like a wiki farm, I agree with the developers:
composer isn't suited for managing MediaWiki extensions.

Let me copy my comment from Github...

On Github, @Ingomueller-net wrote:
So if I understand things correctly, if MediaWiki wanted to use composer, then the correct way would be along the following lines: composer would determine the best version of an extension and install it (i.e., to copy the files from somewhere). The file loaded by the autoloader of composer would register the extension in the extension registry of MediaWiki, but not enable it. Enabling would be done with a different mechanism, for example by calling a function in LocalSettings.php.

...and elaborate: I think that we have everything at hand to fully support your wiki farm use case just with composer only. What is important is that that extensions do not tell composer to autoload a file that runs wfLoadExtensions (although many extensions do load such a file). This should be documented more clearly and possibly tested with a test case. Then the wiki admins would load the desired extensions using wfLoadExtension(s) or require_once "$IP/extensions/ExtensionName/ExtensionName.php" in his/her LocalSettings.php (which would have a switch $_SERVER['REMOTE_HOST'] in your case).

Again in short: I think that everything is ready to use composer, it only has to be used correctly (which needs documentation/help/enforcement).

In T88596#1786388, @bd808 wrote:

I added https://www.mediawiki.org/wiki/Composer#Using_composer-merge-plugin

Great! I understand much better now. The code that is shown is from composer.local.json, right? That could be clearer...

Ingomueller-net writes:

I think that everything is ready to use composer, it only has to be
used correctly (which needs documentation/help/enforcement).

Sounds like something for the developer summit. Will you be there?

Are you willing to make this (doc/help/enforce) happen?

@Legoktm can you tell me what the status is?

In T88596#2016719, @daniel wrote:

@Legoktm can you tell me what the status is?

@Florian has a patch to implement the dependency stuff in extension.json https://gerrit.wikimedia.org/r/#/c/250060/ that I severely owe him CR on. I plan to merge it once the implementation is ready.

That's pretty much it, I've lost most of my interest in this RfC for now, as I'm bored of rehashing the same debates over and over (composer! debs! etc.!). It's been obvious for a while now that no one cares very much about this subject (c.f. https://lists.wikimedia.org/pipermail/wikitech-l/2015-September/083302.html, https://lists.wikimedia.org/pipermail/wikitech-l/2016-January/084450.html), and that the status quo is good enough for most core devs.

That's pretty much it, I've lost most of my interest in this RfC for
now, as I'm bored of rehashing the same debates over and over
(composer! debs! etc.!).

I'm really sorry to hear this. As one of the people who had advocated
for the use of Composer, I went out of my way to talk to the Composer
devs and found that what you said about being unable to work with them
on this was true.

My feeling (from various interactions on this issue in WMF, MediaWiki-Stakeholders-Group, a
couple of SMWCons and other places) was that while core developers and
the WMF itself may not care about extension management, MediaWiki users
were very, very interested in the work being done on extensions and were
unhappy with composer.

The same could be said for the work you've done on configuration. That
sort of thing isn't as interesting to the WMF (where an OPS-heavy
process for managing configuration is already in place), but is to other
MediaWiki users.

I'm still very interested in this topic as well. Extension management is very relevant to anybody trying to maintain a third party wiki.

Anyone interested in driving this RFC? I'm happy to shepherd it if it's active, but if nobody is driving, I'll put it on the "stalled" pile.

Pinging @MarkAHershberger @cicalese @Florian @JanZerebecki @bd808

@daniel: I unfortunately have the same feeling as @Legoktm, compared with the fact, that there seems to be very less input and especially interest in the developer community (which makes reviews and discussions more and more difficult :(). I've also currently doesn't have the time to work on such a big RfC, sorry. If someone else is driving this, I'm happy to support it as much as possible, but that's all I can do currently.

This task is very much in line with the near term plans of the MediaWiki-Stakeholders-Group. Speaking for the MediaWiki-Stakeholders-Group, we would be willing to drive this RFC going forward, if it is possible for a group rather than an individual to do so. If it needs to be an individual, one of us will volunteer. What are the next steps?

@cicalese Cool! There is no clear rule that an RFC has to be driven by an individual. Let's see how it goes. It would be good for me to have some means of contact though.

Next steps would be: propose the next steps for implementing this, possibly even already including code, and discuss them.

Great! Feel free to use me as the point of contact. We'll get working on planning the way forward.

Two more immediate things I know of that need doing here: 1) Getting the open patch for T117277 merged. 2) Implement some sort of declaration of optional dependencies.
Both together will cover everything that is currently only in dependencies in integration/config.git/zuul/parameter_functions.py . So it would be useful to then make the CI obey extension.json instead of its own dependency information.

@cicalese Any progress on this? Do you have all the information you need?

It seems like the next concrete step would be to finalize If1cccee1a16. Want to chime in on that?

@daniel, I'm sorry for the delayed reply. We had some initial discussion about this in a recent MediaWiki Stakeholder Group meeting, and we need to expand upon that discussion and come up with a plan. I hope we can have some further discussion when some of us are together at the upcoming EMWCon.

Ok. I'll mark it as stalled for now. Fell free to move it to the RFC inbox or ping me when you have a plan for this.

Mentioning this RfC as I think it's relevant to this task.

https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_feedback

Thank you for hosting your session at WMHack! If you have any notes or slides please add them to the task and then make sure to close the task when there are no more actions. :)

This has been superseded by https://www.mediawiki.org/wiki/Extension_management_2018_feedback and T199372.
Neither I nor @Legoktm see a point in pushing this through the RFC process.

Closing.