This should probably be answered by @bd808 or @Legoktm .

In the long run not running composer in projects that use composer for dependencies is probably a bad idea. As is duplicate declaration of dependencies in two places without something that automatically synchronizes them. Anyway that seems more like a long run thing, so unless we can switch to use something nice for the next deployment (I don't follow the extension registration and composer stuff in Mediawiki closely enough to know), we should find a solution that works now:

Wikidata CI works by running composer as one of the first things in each job, deviating from that for one new Wikidata extension seems like a bad idea, unless we are ready to change this for everything.

In T88436#1011709, @JanZerebecki wrote:

This should probably be answered by @bd808 or @Legoktm .

In the long run not running composer in projects that use composer for dependencies is probably a bad idea. As is duplicate declaration of dependencies in two places without something that automatically synchronizes them. Anyway that seems more like a long run thing, so unless we can switch to use something nice for the next deployment (I don't follow the extension registration and composer stuff in Mediawiki closely enough to know), we should find a solution that works now:

Wikidata CI works by running composer as one of the first things in each job, deviating from that for one new Wikidata extension seems like a bad idea, unless we are ready to change this for everything.

The mediawiki/vendor repository is for Composer manged libraries that are needed to support MediaWiki and the extensions installed on the WMF production cluster. Wikidata was using Composer prior to the creation of this repository by creating their own bundle of Composer managed libraries inside the "built" mediawiki/extensions/Wikidata repo. I'm not opposed at all to Wikidata moving to a model where they place their dependencies in mediawiki/vendor instead of an internal vendor collection. In the long term this will be good actually as it will make the WMF cluster more explicit about shared dependencies. Today if multiple extensions each ship their own version of class Foo the actual version of Foo loaded at runtime for a given request is dependent on the installation order of the duplicating extensions and how they expose Foo to the autoloader.

But manually copying the dependencies of more than a dozen components (Wikibase alone has more direct dependencies) and even recursively for their dependencies does not seem a good way to improve this. The current build process is fully automated. So in the long run something like https://github.com/wikimedia/composer-merge-plugin seems like a better idea than manually doing this. But even then we would probably run composer in the CI of individual extensions and only replace it by a build when testing release branches, as otherwise we can't test changes to the dependencies before making a building with that change. Or is there something i'm missing that would allow us to not run composer?

+1 for using https://github.com/wikimedia/composer-merge-plugin or such to pull in wikibase extensions + our libraries into the core installation for deployment, instead of making our own build.

In T88436#1012436, @JanZerebecki wrote:

But manually copying the dependencies of more than a dozen components (Wikibase alone has more direct dependencies) and even recursively for their dependencies does not seem a good way to improve this.

Why would you need to do it recursively? Just include Wikibase's dependencies in vendor (still duplicated manually :() and let composer automatically figure out their dependencies.

The current build process is fully automated. So in the long run something like https://github.com/wikimedia/composer-merge-plugin seems like a better idea than manually doing this. But even then we would probably run composer in the CI of individual extensions and only replace it by a build when testing release branches, as otherwise we can't test changes to the dependencies before making a building with that change. Or is there something i'm missing that would allow us to not run composer?

Right now extensions are tested with mediawiki/vendor loaded, so any dependency in there can be used when running tests. I had talked with @Krinkle yesterday about something related and he suggested having CI run composer install during normal commits, and wmf/* branch ones would use mediawiki/vendor.

extensions/Wikidata/composer.json has the following dependencies:

• wikibase/Wikidata.org (https://github.com/wmde/Wikidata.org) - an extension with no dependencies that could be moved to gerrit and deployed in the normal method
• wikibase/wikimedia-badges (https://github.com/wmde/WikimediaBadges) - same
• propertysuggester/property-suggester (https://github.com/Wikidata-lib/PropertySuggester) - same, though you would lose the composer autoloader
• wikibase/wikibase (the "Wikibase" extension) - which has 19 dependencies

So really only Wikibase has dependencies that need to be added to and managed with mediawiki/vendor.

I propose we stop using mediawiki-vendor for the majority of MediaWiki core and extension repositories Jenkins jobs (master and release branches).

In the wmf-branches (of MediaWiki core and extensions) we'd use mediawiki-vendor instead of composer. (And the same for the master branch of mediawiki-vendor itself).

The commits that create or modify wmf branches will then run with mediawiki-vendor and thus ensure (before deployment!) that everything is in order. This means that (if someone forgets to make a commit to mediawiki-vendor) issues are caught a bit later. I think that's worth the reduced overall complexity and gained flexibility for non-wmf extensions. It also makes it a lot easier to write commits against mediawiki/core that update composer.json. It'll enable developers to test these commits in Jenkins – without having to orchestrate inter-dependent commits and merge them before testing.

This also in light of the recent issue with update.php verifying the composer dependencies. See T88211: Unable to update libraries in MediaWiki core and vendor due to version mismatch and circular dependencies.

We update dependency versions quite often for Wikidata, so we depend on Jenkins testing against the current dependencies specified in composer.json.

In T88436#1012609, @Legoktm wrote:

Why would you need to do it recursively? Just include Wikibase's dependencies in vendor (still duplicated manually :() and let composer automatically figure out their dependencies.

Yes you are right no need for doing it recursively there.

In T88436#1013289, @Krinkle wrote:

I propose we stop using mediawiki-vendor for the majority of MediaWiki core and extension repositories Jenkins jobs (master and release branches).

In the wmf-branches (of MediaWiki core and extensions) we'd use mediawiki-vendor instead of composer. (And the same for the master branch of mediawiki-vendor itself).

I think that is a good idea.

In the long run it might be a good idea to make mediawiki-vendor a build result of the composer run in wmf branches of mediawiki-core and run the tests after that with the newly built vendor repo.

@hashar What do you think? Should we move forward in the outlined direction and merge the linked gerrit patch?

In T88436#1025151, @JanZerebecki wrote:

In the long run it might be a good idea to make mediawiki-vendor a build result of the composer run in wmf branches of mediawiki-core and run the tests after that with the newly built vendor repo.

As currently envisioned, mediawiki/vendor is a very closely controlled repository. This is code that we execute on the WMF production cluster and as such changes need to be vetted by humans to ensure that only trusted code is allowed in. If we started making this an automated repo that is not vetted via gerrit review we would run the risk of a malicious upstream inserting undesired code into the production runtime environment.

I forgot for a moment that core pulls in externally maintained dependencies via composer. I distracted from the main question for this task:

Should we run composer for master branches (i.e. not the wmf deployment ones) during CI instead of relying on mediawiki-vendor?

In T88436#1025775, @JanZerebecki wrote:

Should we run composer for master branches (i.e. not the wmf deployment ones) during CI instead of relying on mediawiki-vendor?

Yes. Both to simplify day-to-day maintenance (e.g. atomic changes that bump dependencies and update calling code) for production dependencies. As well as for the ability to use devDependencies for e.g. phpcs and phpunit. Which is already used by projects not tied to MediaWiki.

Jobs will no longer use mediawiki/vendor in the near future. But until we're there, if the relevant extension here is Wikimedia-deployed and needs data-values/javascript at run-time in production, then it not only can be added but must be added.

But until we're there, if the relevant extension here is Wikimedia-deployed and needs data-values/javascript at run-time in production, then it not only can be added but must be added.

Both this extension and its dependencies (data-values/javascript) are only pulled into the wikidata build. So is there anything against merging that proposed patch?

In T88436#1042729, @Krinkle wrote:

Jobs will no longer use mediawiki/vendor in the near future. But until we're there, if the relevant extension here is Wikimedia-deployed and needs data-values/javascript at run-time in production, then it not only can be added but must be added.

As @JanZerebecki mentioned, this is already Wikimedia-deployed via the Wikidata build. For production, there is currently no need to do the proposed addition (although it might be a lot simpler and cleaner to remove the build and add the dependencies to mediawiki/vendor instead). For CI, this change is imho a bad idea, since CI should run against the current dependencies stated in composer.json.

In T88436#1013289, @Krinkle wrote:

I propose we stop using mediawiki-vendor for the majority of MediaWiki core and extension repositories Jenkins jobs (master and release branches).

Filed T90303 for doing this.