What's the benefit over just using Packagist? Operational security?

In T170897#3546595, @Tgr wrote:

What's the benefit over just using Packagist? Operational security?

Using Packagist would require that every extension has a valid composer.json file as well as submitting the package to Packagist.

We would also need to implement a webhook on our side to inform Packagist of new versions (they have one for GitHub, but not Gerrit/Phabricator that I am aware of).

Using Packagist would require that every extension has a valid composer.json file as well as submitting the package to Packagist.

Yes but that seems like the right thing to do anyway. An external party trying to maintain the composer file for a developer who doesn't care themselves would probably end in outdated package descriptions very quickly. Drupal doesn't do that either; they require the module owner to maintain composer.json. (Also Drupal uses their own package repo to select the module version matching the Drupal core version. That is clever but not really applicable to MediaWiki.)

We would also need to implement a webhook on our side to inform Packagist of new versions (they have one for GitHub, but not Gerrit/Phabricator that I am aware of).

That seems both less effort than running our own package repository and more generally useful (surely there are people who want to host their extension in gerrit but want to make it available via packagist.org).

IMO it would be more useful to stick with Packagist and creating composer.json best practices for extension/skin developers would (the current document is an ungodly mess).

In T170897#3546697, @Tgr wrote:

Yes but that seems like the right thing to do anyway.

Well I certainly agree. :) But we would need to make a policy and automate the process (for extensions hosted on wikimedia).

Drupal doesn't do that either; they require the module owner to maintain composer.json.

Drupal doesn't even require that a module has a composer.json at all. It just pulls the necessary data from the module's .info.yml file if a value they need is missing.

However, the description is required by Packagist, not by Composer. So we wouldn't need to have it in our own Packagist at all.

Drupal also has non W.X.Y.Z version numbers which are also incompatible with Packagist.

In our case, we could parse extension.json with what is needed if it isn't provided by the the module's composer.json (if there is one).

Then again, if we are automating creating/updating composer.json we could just do that for the extensions and then push them to the official Packagist.

That seems both less effort than running our own package repository and more generally useful (surely there are people who want to host their extension in gerrit but want to make it available via packagist.org).

IMO it would be more useful to stick with Packagist and creating composer.json best practices for extension/skin developers would (the current document is an ungodly mess).

I completely agree, but getting people on board with that has proven extremely difficult thus far.

@Tgr see T167329#3444793

In T170897#3546609, @dbarratt wrote:

We would also need to implement a webhook on our side to inform Packagist of new versions (they have one for GitHub, but not Gerrit/Phabricator that I am aware of).

All gerrit repos are mirrored to GitHub where the Packagist hook can be enabled, so this one is trivially handled.

The million dollar question that all of these fragmented tickets is not directly addressing is "should MediaWiki extensions and skins be deployed/installed/enabled with Composer at all." @dbarratt has some good arguments for doing this I think, but I'd really like to see an RFC written, discussed, and approved before we put any effort into various bits and pieces of Composer/Packagist support for non-library components.

Once we start doing something like running a repository it will be used by some non-zero number of people. That then means that some non-zero number of users will have their deployments and workflows broken if the service is down or decommissioned. That in turn implies that there needs to be some sort of service level agreement (SLA) about the service from the start and some group of 2 or more people who are committed to keeping the service in good running order. These things always seem like cheap experiments when they start, but 18 months later often they have become a dragging burden on the limited technical resources of the movement with unclear return on that investment.

In T170897#3546804, @bd808 wrote:

The million dollar question that all of these fragmented tickets is not directly addressing is "should MediaWiki extensions and skins be deployed/installed/enabled with Composer at all." @dbarratt has some good arguments for doing this I think, but I'd really like to see an RFC written, discussed, and approved before we put any effort into various bits and pieces of Composer/Packagist support for non-library components.

Once we start doing something like running a repository it will be used by some non-zero number of people. That then means that some non-zero number of users will have their deployments and workflows broken if the service is down or decommissioned. That in turn implies that there needs to be some sort of service level agreement (SLA) about the service from the start and some group of 2 or more people who are committed to keeping the service in good running order. These things always seem like cheap experiments when they start, but 18 months later often they have become a dragging burden on the limited technical resources of the movement with unclear return on that investment.

Valid.

I can certainly write that up as an RFC, but I think I should be clear that "deploying/installing/enabling" is taking it too far. At this point we're talking about only loading an extension into the file system by using Composer's CLI (which is impossible right now).

If we want to talk about "deploying/installing/enabling", that is a much larger discussion (that already had a declined RFC) and I think that puts the cart well before the horse.

@dbarratt I would personally consider "loading an extension into the file system by using Composer's CLI" and "deploying" to be synonyms. You are downloading and unpacking a tarball/zipfile via an automated process and likely doing so with the expectation that later you can update that unpacked directory tree to match upstream changes. This then sets up the a fore mentioned expectation that said services will function at a later date and the associated maintenance burden for the technical community.

Today we have two well supported and documented methods for loading an extension into the file system: git clones and tarballs. Both provide versioned content. Both have support systems with maintainers who respond to both systemic failures and individual user bug reports. Neither really has an explicit SLA, but history has shown an implicit SLA with reasonably high availability. Of the two, tarballs is actually the less supported. If and when @Legoktm gets too oversubscribed to keep that service running the technical community will need to scramble to replace his stewardship. I'm not currently convinced that adding a 3rd download method is a vast improvement to end-user usability. I'm open to hearing why this is a net win over other methods and how it will increase the overall adoption of MediaWiki however. This is what I think an RFC should be addressing first and foremost.

By tagging with Release-Engineering-Team this task implies that that specific Foundation team should be responsible for deploying and maintaining the service. I'm not the manager of that team (hi @greg!), but I'm pretty familiar with their mission, workload, and staffing. If you are going to convince them to support this, you are going to need to some armed with more evidence of how this will make the movement stronger and ideally what existing sub-standard service(s) it can replace.

In T170897#3546890, @bd808 wrote:

@dbarratt I would personally consider "loading an extension into the file system by using Composer's CLI" and "deploying" to be synonyms. You are downloading and unpacking a tarball/zipfile via an automated process and likely doing so with the expectation that later you can update that unpacked directory tree to match upstream changes. This then sets up the a fore mentioned expectation that said services will function at a later date and the associated maintenance burden for the technical community.

Valid.

Today we have two well supported and documented methods for loading an extension into the file system: git clones and tarballs. Both provide versioned content. Both have support systems with maintainers who respond to both systemic failures and individual user bug reports. Neither really has an explicit SLA, but history has shown an implicit SLA with reasonably high availability. Of the two, tarballs is actually the less supported. If and when @Legoktm gets too oversubscribed to keep that service running the technical community will need to scramble to replace his stewardship. I'm not currently convinced that adding a 3rd download method is a vast improvement to end-user usability. I'm open to hearing why this is a net win over other methods and how it will increase the overall adoption of MediaWiki however. This is what I think an RFC should be addressing first and foremost.

Understood.

Unrelated question... why don't we use git tagging? This I think would ease tarball creation (github creates them automatically) and would be a step towards good Composer support as well.

We do use git tags for things that are deployed to the Foundation wikis:

mbp01:~/projects/wmf/vagrant/mediawiki  (git master $)
bd808$ git tag
1.1.0
1.10.0
1.10.0rc1
1.10.0rc2
(... snip ...)
wmf/1.26wmf7
wmf/1.26wmf8
wmf/1.26wmf9

Tagging for arbitrary extensions/skins is up to the maintainers of those extensions/skins.

In T170897#3546890, @bd808 wrote:

By tagging with Release-Engineering-Team this task implies that that specific Foundation team should be responsible for deploying and maintaining the service. I'm not the manager of that team (hi @greg!), but I'm pretty familiar with their mission, workload, and staffing. If you are going to convince them to support this, you are going to need to some armed with more evidence of how this will make the movement stronger and ideally what existing sub-standard service(s) it can replace.

And to be explicit: us moving tasks into our "watching / external" project means that we believe we are not drivers for the task but are willing to be consulted (or we're just curious and watching). In this specific case: we are not aware of any direct need nor consensus to do any of this work so we'll wait until that changes to review our role in this. And as @bd808 said: we have other competing priorities so a stronger argument will be needed ;)

In T170897#3546890, @bd808 wrote:

Today we have two well supported and documented methods for loading an extension into the file system: git clones and tarballs. [...] Of the two, tarballs is actually the less supported.

I'm surprised to hear that Git install would be favoured over the tarball download, because I don't think there's even a direct link to the cloneable URL in most extension pages. Oh, no I guess you mean that it's more supported which I guess is just that there is more support burden on the ExtensionDistributor, which makes sense.

Anyway, what I was going to say, as someone who both regularly installs extensions with Composer and who maintains an extension explicitly installable this way, is that it seems to me that it's really not very hard for extension maintainers who want to use Composer to do so. The main missing link is the auto-update for Packagist.

Basically, an extension

• has to have a full composer.json file (which in most cases means adding only name, description, type, and license; none of which change very often) and then be registered on Packagist;
• by registered under individual maintainers' names;
• needs to be tagged — but that's nothing really to do with this discussion is it? versioned software that isn't tagged is just lazy! :)
• Packagist needs to be manually updated whenever there's a new version (this can be done with a one-line curl script);
• Installing sysadmins then add the extension to their composer.local.json as a requirement, and update.

Everything seems to just work, for me. The main arguments against this that I've heard is that it's hard for wikifarms who are running multiple MediaWiki versions and/or who want to selectively enable extensions on different wikis. I think the latter held true in the days before extension registration, because the extension entry point would be autoloaded (and nothing added to LocalSettings.php) but it's not the case any more I think? I'm not sure about the former, maybe there's something I'm not thinking about.

(In fact, if the CI process had a way of maintainers adding private environment variables (like Travis does for instance) then the updating packagist part could be done electively by whoever wants to do it.)

So, yeah, I love the idea of just using Packagist; less to support, more visibility, more normal, and so much easier to keep a wiki up to date. :-)

Back in 2015 when I wrote https://www.mediawiki.org/wiki/Requests_for_comment/Improving_extension_management I investigated using composer/packagist. There were some reasons I listed on the RfC that are no longer relevant (e.g. lack of distro packaging).

But I think the main issue is that conceptually extensions really don't fit the composer model. Most extensions aren't tagged, nearly all use a rolling release model which may or may not correspond to the release branches (ExtensionDistributor provides a "best effort" implementation in this case).

I wrote up a separate proposal for a small service that would do basically what is being proposed here but with MediaWiki-specific knowledge: https://www.mediawiki.org/wiki/Extension:ExtensionDistributor/tardist

I think the biggest thing we need right now (and had favorable support at the time) was a command line extension manager (enable/disable/download/update/etc.) POC: https://gerrit.wikimedia.org/r/#/c/308891/ There actually used to be one in MediaWiki core but it was removed because it sucked. I'm not yet convinced that composer is the solution for that. I'm sure we can get it to work somehow, but whether it's actually the correct solution I'm doubtful.

In T170897#3547492, @Legoktm wrote:

But I think the main issue is that conceptually extensions really don't fit the composer model.

Can you explain this? I think the only difference between an extension and a PHP package is that it goes in ./extensions rather than ./vendor and Composer Installers handles this for us (just like it handles it for Drupal)

Most extensions aren't tagged, nearly all use a rolling release model which may or may not correspond to the release branches (ExtensionDistributor provides a "best effort" implementation in this case).

Composer supports rolling releases, so that isn't really an issue. But if an extension has versions, we should encourage (by policy?) extensions to tag the versions on Git.

I wrote up a separate proposal for a small service that would do basically what is being proposed here but with MediaWiki-specific knowledge: https://www.mediawiki.org/wiki/Extension:ExtensionDistributor/tardist

But this doesn't use Composer correct?

I think the biggest thing we need right now (and had favorable support at the time) was a command line extension manager (enable/disable/download/update/etc.) POC: https://gerrit.wikimedia.org/r/#/c/308891/ There actually used to be one in MediaWiki core but it was removed because it sucked. I'm not yet convinced that composer is the solution for that. I'm sure we can get it to work somehow, but whether it's actually the correct solution I'm doubtful.

Can Composer just be the solution for download and update right now? Then a different CLI can be used for enabling and disabling?

I ask because that's what Drupal does. There is no enabling and disabling with Composer (on purpose). And the extensions in Drupal do not get autoloaded with Composer. The only thing you can do (and some would argue ought to do) with Composer is download and update an extension with all of it's dependencies (in a recursive fashion).

I really do not see the point in re-inventing the wheel on downloading/uploading extensions with recursive dependency resolution.

We could either make this happen with a technology solution (our own Composer repository) or via policy (which could be enforced with the CI). I favor the policy solution as it's a whole lot less work overall on everyone.