Page MenuHomePhabricator

Add data-values/javascript 0.6.0 to mediawiki/vendor
Closed, ResolvedPublic

Description

Several extensions depend on the PHP library  data-values/javascript. Adrian is working on the WikibaseJavaScriptApi extension and rely on composer.json to fetch it.

The CI jobs running extensions tests and qunit do not use composer in extensions but expect the dependencies to be shipped via mediawiki/vendor. Adrian requested the jobs via T86176 and even went through the trouble of creating more specific jobs in Jenkins https://gerrit.wikimedia.org/r/#/c/180418 .

I know from past experience this open a whole new can of worm that would cause extensions developers to add more and more libraries to their extensions which will end up being a dependency hell when we want to deploy them.

As I understand it, mediawiki/vendor is kind of a global requirement / dependency list for all extensions to be deployed on Wikimedia cluster. Thus I propose to amend it with data-values/javascript .

The Jenkins job roughly does:

git clone mediawiki/core
git clone mediawiki/vendor
php maintenance/install.php
git clone mediawiki/extensions/WikibaseJavaScriptApi
php maintenance/update.php
# "unit" tests
cd tests/phpunit && php phpunit.php --testsuite extensions
# qunit tests
point a browser at [[Special:JavascriptTest/qunit]]

I am assuming shipping datavalues/javascript in mediawiki/vendor is sufficient to expose it to extensions relying on it.

Side effects: Wikibase is deployed using a custom repository which use composer to ship their libraries. data-values/javascript will probably need to be removed from it since it would be shipped by mediawiki/vendor.

Event Timeline

hashar created this task.Feb 3 2015, 3:45 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2015, 3:45 PM

This should probably be answered by @bd808 or @Legoktm .

In the long run not running composer in projects that use composer for dependencies is probably a bad idea. As is duplicate declaration of dependencies in two places without something that automatically synchronizes them. Anyway that seems more like a long run thing, so unless we can switch to use something nice for the next deployment (I don't follow the extension registration and composer stuff in Mediawiki closely enough to know), we should find a solution that works now:

Wikidata CI works by running composer as one of the first things in each job, deviating from that for one new Wikidata extension seems like a bad idea, unless we are ready to change this for everything.

bd808 added a comment.Feb 3 2015, 6:00 PM

This should probably be answered by @bd808 or @Legoktm .
In the long run not running composer in projects that use composer for dependencies is probably a bad idea. As is duplicate declaration of dependencies in two places without something that automatically synchronizes them. Anyway that seems more like a long run thing, so unless we can switch to use something nice for the next deployment (I don't follow the extension registration and composer stuff in Mediawiki closely enough to know), we should find a solution that works now:
Wikidata CI works by running composer as one of the first things in each job, deviating from that for one new Wikidata extension seems like a bad idea, unless we are ready to change this for everything.

The mediawiki/vendor repository is for Composer manged libraries that are needed to support MediaWiki and the extensions installed on the WMF production cluster. Wikidata was using Composer prior to the creation of this repository by creating their own bundle of Composer managed libraries inside the "built" mediawiki/extensions/Wikidata repo. I'm not opposed at all to Wikidata moving to a model where they place their dependencies in mediawiki/vendor instead of an internal vendor collection. In the long term this will be good actually as it will make the WMF cluster more explicit about shared dependencies. Today if multiple extensions each ship their own version of class Foo the actual version of Foo loaded at runtime for a given request is dependent on the installation order of the duplicating extensions and how they expose Foo to the autoloader.

But manually copying the dependencies of more than a dozen components (Wikibase alone has more direct dependencies) and even recursively for their dependencies does not seem a good way to improve this. The current build process is fully automated. So in the long run something like https://github.com/wikimedia/composer-merge-plugin seems like a better idea than manually doing this. But even then we would probably run composer in the CI of individual extensions and only replace it by a build when testing release branches, as otherwise we can't test changes to the dependencies before making a building with that change. Or is there something i'm missing that would allow us to not run composer?

aude added a subscriber: aude.Feb 3 2015, 8:17 PM

+1 for using https://github.com/wikimedia/composer-merge-plugin or such to pull in wikibase extensions + our libraries into the core installation for deployment, instead of making our own build.

Legoktm added a subscriber: Krinkle.Feb 3 2015, 8:42 PM

But manually copying the dependencies of more than a dozen components (Wikibase alone has more direct dependencies) and even recursively for their dependencies does not seem a good way to improve this.

Why would you need to do it recursively? Just include Wikibase's dependencies in vendor (still duplicated manually :() and let composer automatically figure out their dependencies.

The current build process is fully automated. So in the long run something like https://github.com/wikimedia/composer-merge-plugin seems like a better idea than manually doing this. But even then we would probably run composer in the CI of individual extensions and only replace it by a build when testing release branches, as otherwise we can't test changes to the dependencies before making a building with that change. Or is there something i'm missing that would allow us to not run composer?

Right now extensions are tested with mediawiki/vendor loaded, so any dependency in there can be used when running tests. I had talked with @Krinkle yesterday about something related and he suggested having CI run composer install during normal commits, and wmf/* branch ones would use mediawiki/vendor.

extensions/Wikidata/composer.json has the following dependencies:

So really only Wikibase has dependencies that need to be added to and managed with mediawiki/vendor.

Legoktm set Security to None.

I propose we stop using mediawiki-vendor for the majority of MediaWiki core and extension repositories Jenkins jobs (master and release branches).

In the wmf-branches (of MediaWiki core and extensions) we'd use mediawiki-vendor instead of composer. (And the same for the master branch of mediawiki-vendor itself).

The commits that create or modify wmf branches will then run with mediawiki-vendor and thus ensure (before deployment!) that everything is in order. This means that (if someone forgets to make a commit to mediawiki-vendor) issues are caught a bit later. I think that's worth the reduced overall complexity and gained flexibility for non-wmf extensions. It also makes it a lot easier to write commits against mediawiki/core that update composer.json. It'll enable developers to test these commits in Jenkins – without having to orchestrate inter-dependent commits and merge them before testing.

This also in light of the recent issue with update.php verifying the composer dependencies. See T88211: Unable to update libraries in MediaWiki core and vendor due to version mismatch and circular dependencies.

We update dependency versions quite often for Wikidata, so we depend on Jenkins testing against the current dependencies specified in composer.json.

Why would you need to do it recursively? Just include Wikibase's dependencies in vendor (still duplicated manually :() and let composer automatically figure out their dependencies.

Yes you are right no need for doing it recursively there.

I propose we stop using mediawiki-vendor for the majority of MediaWiki core and extension repositories Jenkins jobs (master and release branches).
In the wmf-branches (of MediaWiki core and extensions) we'd use mediawiki-vendor instead of composer. (And the same for the master branch of mediawiki-vendor itself).

I think that is a good idea.

Lydia_Pintscher triaged this task as Medium priority.Feb 8 2015, 11:17 AM
Lydia_Pintscher added a project: Wikidata.
Lydia_Pintscher added a subscriber: Lydia_Pintscher.

In the long run it might be a good idea to make mediawiki-vendor a build result of the composer run in wmf branches of mediawiki-core and run the tests after that with the newly built vendor repo.

@hashar What do you think? Should we move forward in the outlined direction and merge the linked gerrit patch?

bd808 added a comment.Feb 9 2015, 4:52 PM

In the long run it might be a good idea to make mediawiki-vendor a build result of the composer run in wmf branches of mediawiki-core and run the tests after that with the newly built vendor repo.

As currently envisioned, mediawiki/vendor is a very closely controlled repository. This is code that we execute on the WMF production cluster and as such changes need to be vetted by humans to ensure that only trusted code is allowed in. If we started making this an automated repo that is not vetted via gerrit review we would run the risk of a malicious upstream inserting undesired code into the production runtime environment.

I forgot for a moment that core pulls in externally maintained dependencies via composer. I distracted from the main question for this task:

Should we run composer for master branches (i.e. not the wmf deployment ones) during CI instead of relying on mediawiki-vendor?

Should we run composer for master branches (i.e. not the wmf deployment ones) during CI instead of relying on mediawiki-vendor?

Yes. Both to simplify day-to-day maintenance (e.g. atomic changes that bump dependencies and update calling code) for production dependencies. As well as for the ability to use devDependencies for e.g. phpcs and phpunit. Which is already used by projects not tied to MediaWiki.

Jobs will no longer use mediawiki/vendor in the near future. But until we're there, if the relevant extension here is Wikimedia-deployed and needs data-values/javascript at run-time in production, then it not only can be added but must be added.

But until we're there, if the relevant extension here is Wikimedia-deployed and needs data-values/javascript at run-time in production, then it not only can be added but must be added.

Both this extension and its dependencies (data-values/javascript) are only pulled into the wikidata build. So is there anything against merging that proposed patch?

Jobs will no longer use mediawiki/vendor in the near future. But until we're there, if the relevant extension here is Wikimedia-deployed and needs data-values/javascript at run-time in production, then it not only can be added but must be added.

As @JanZerebecki mentioned, this is already Wikimedia-deployed via the Wikidata build. For production, there is currently no need to do the proposed addition (although it might be a lot simpler and cleaner to remove the build and add the dependencies to mediawiki/vendor instead). For CI, this change is imho a bad idea, since CI should run against the current dependencies stated in composer.json.

I propose we stop using mediawiki-vendor for the majority of MediaWiki core and extension repositories Jenkins jobs (master and release branches).

Filed T90303 for doing this.

JanZerebecki closed this task as Resolved.Apr 14 2015, 3:02 PM
JanZerebecki claimed this task.

The patch is merged future work is T90303 and T95663.

bd808 moved this task from Untriaged to Done on the Librarization board.Jul 22 2015, 11:16 PM