Page MenuHomePhabricator

[Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor
Closed, DeclinedPublic

Description

eventually there will be some component used by more than one extension. these dependencies should be managed centrally for wmf's mediawiki installation instead of putting everything into our own build.

https://github.com/wikimedia/composer-merge-plugin

https://github.com/wikimedia/mediawiki-vendor

this means still having WikidataBuildResources but not run composer install in it and maybe rename / have it be the "Wikidata" extension that we deploy.

https://github.com/wikimedia/mediawiki-extensions-Wikidata

the one problem with this is that we have some dependencies that are installed as extensions, in "$IP/extensions/...". It is weird and probably problematic to do that with composer, side-by-side with the other extensions that are handled as submodules. I am not sure where would put them instead or how we would handle this.

The extensions that currently get installed via composer in WikidataBuildResources need to then become submodules of the wmf branches of core.

Current Wikidata deployment process is documented at https://wikitech.wikimedia.org/wiki/How_to_deploy_Wikidata_code.

Related Objects

StatusAssignedTask
ResolvedAddshore
ResolvedAddshore
InvalidNone
DeclinedNone
Resolvedaude
ResolvedJeroenDeDauw
ResolvedWMDE-leszek
InvalidNone
InvalidNone
OpenNone
ResolvedVictorbarbu
ResolvedLucas_Werkmeister_WMDE
ResolvedMtDu
ResolvedLydia_Pintscher
DeclinedLydia_Pintscher
InvalidNone
InvalidNone
ResolvedReedy
ResolvedLegoktm
ResolvedLadsgroup
ResolvedNone
ResolvedJanZerebecki
OpenNone
ResolvedJanZerebecki
ResolvedKrinkle
InvalidNone
ResolvedJanZerebecki
ResolvedAddshore

Event Timeline

aude created this task.Apr 10 2015, 9:20 AM
aude raised the priority of this task from to Normal.
aude updated the task description. (Show Details)
aude added a project: Wikidata.
aude added a subscriber: aude.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 10 2015, 9:20 AM
aude added a comment.Apr 10 2015, 9:23 AM

for myself, T94672 was caused by having WikibaseDataModel in two places, one apparently using master or a more recent version (which is the one that got used in my install) vs the one in Wikibase.

this just silently happened :o with no indication of a version conflict or anything and could be quite problematic to have this happen in production somewhere.

In the Wikidata entry point, we have:

if ( !empty( $wmgUseWikibaseRepo ) ) {                                                                 
    include_once __DIR__ . '/extensions/Wikibase/repo/Wikibase.php';                                   
    include_once __DIR__ . '/extensions/Wikidata.org/WikidataOrg.php';                                 
    include_once __DIR__ . '/extensions/PropertySuggester/PropertySuggester.php';                      
    include_once __DIR__ . '/WikibaseRepo.settings.php';                                               
}                                                                                                      
                                                                                                       
if ( !empty( $wmgUseWikibaseClient ) ) {                                                               
    include_once __DIR__ . '/extensions/Wikibase/client/WikibaseClient.php';                           
    include_once __DIR__ . '/WikibaseClient.settings.php';                                             
}

this won't work and not flexible for composer-merge-plugin + the current build approach

awight added a subscriber: awight.Apr 23 2015, 9:28 AM
Jonas renamed this task from Use composer-merge-plugin to include Wikidata components in mediawiki-vendor to [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor.Aug 13 2015, 3:00 PM
awight removed a subscriber: awight.Nov 17 2015, 11:13 AM
hoo claimed this task.Jan 9 2017, 9:36 PM
hoo reassigned this task from hoo to demon.
hoo added a project: Release-Engineering-Team.
hoo added subscribers: Addshore, Legoktm, JeroenDeDauw and 3 others.
hoo added a comment.Mar 8 2017, 4:20 PM

Any updates here @demon? Is there anything we can/ should do to get this going?

I just simply haven't had the time to look into it yet, I apologize.

I don't think there's any need to use the composer-merge-plugin here...if something is a MW extension, it should be under the mw/extensions/ hierarchy in Gerrit, and brought in through the normal deployment process. If it's a library, it should be published on packagist and added to mediawiki/vendor like all other libraries are.

If it's a library, it should be published on packagist and added to mediawiki/vendor like all other libraries are.

Well, that's mostly the thought here, but the task summary may be a little unclear (this was born out of a conversation @hoo and I have at the dev summit)

hoo added a subscriber: greg.Apr 12 2017, 11:47 AM

@demon @greg Any updates here? Is there anything we can help you with?

Reedy added a subscriber: Reedy.Apr 12 2017, 12:24 PM

Just add the stuff you need to mediawiki-vendor? :) Presumably everything that's in Wikidata/vendor should just go into mediawiki-vendor. Not tracking dev-master though, because that will mean random people doing composer update will get random updates that they won't know if they should be committed

It might be easier if you've some sort of tracking package, so you don't have to update each library individually in the vendor repo to go with the bumps... So along with your bumps to the release tool, you make the changes to mediawiki-vendor, and get them submitted before the branch cut too

And using composer-merge-plugin would only make sense if we were building the vendor on the fly during the deployment process, as it could then crawl and include as necessary..

Expecting everyone to have the same dev setup so that a composer.local.json type file that pointed to Wikidata being in the right place...

{
	"extra": {
		"merge-plugin": {
			"include": [
				"extensions/example/composer.json"
			]
		}
	}
}
hoo added a comment.Apr 19 2017, 7:59 AM

@Reedy We have our "dependency template" in the wikidata/build-resources repo. It's mostly just straight-forward pulling things in via composer, but we also do some evil-ish magic in build/tasks/updatecomposer.js.

Also we have two scripts running via composer (also in the wikidata/build-resources repo):

"scripts": {
    "post-install-cmd": "Wikidata\\SettingsFileGenerator::generateDefaultSettings",
    "post-update-cmd": "Wikidata\\SettingsFileGenerator::generateDefaultSettings"
},
hoo added a comment.Apr 28 2017, 8:08 AM

Any updates here? For us it's good enough if we can manually update the vendor "build" when needed, we don't need any automatism here that reacts upon Wikibase wmf-branch changes.

Reedy added a comment.Apr 28 2017, 4:00 PM

Can't see much reason that you can't just do that...

Krinkle removed a subscriber: Krinkle.Apr 29 2017, 2:45 AM
hoo updated the task description. (Show Details)May 21 2017, 12:31 PM
hoo updated the task description. (Show Details)

The initial step is composer dependencies, so it sounds like all wikidata extensions's composer dependencies need to be tracked somehow in mediawiki/vendor.

Next, Wikidata extensions are not submodules of Wikidata master, so we need to also figure out how to track extensions defined in build-resources as mentioned above.

Also, need to ensure that all the composer post-install and post-update scripts are run.

It's definitely a lot to automate to get everything in sync.

demon removed demon as the assignee of this task.Jun 14 2017, 5:34 AM
demon added a subscriber: demon.

@Addshore and I have been discussing this lately and he uploaded a giant proof of concept to add the Wikidata dependencies to mediawiki/vendor: https://gerrit.wikimedia.org/r/#/c/364712/2

I think we should start this by doing it one extension/library at a time, starting with those that aren't modified as often. Libraries should go in mediawiki/vendor (though there's still a dependency listed in Wikibase's composer.json) instead of the "Wikidata" extension, and sub-extensions should be loaded like a normal extension through mediawiki-config, etc.

I think we should start this by doing it one extension/library at a time, starting with those that aren't modified as often. Libraries should go in mediawiki/vendor (though there's still a dependency listed in Wikibase's composer.json) instead of the "Wikidata" extension, and sub-extensions should be loaded like a normal extension through mediawiki-config, etc.

I am more than happy to try and figure out the 'correct' order for putting these in mediawiki/vendor and creating the needed patches. But perhaps once the big proof on concept patch passes jenkins (we are chasing up those issues now)

Addshore closed this task as Declined.Dec 1 2017, 7:15 PM
Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptDec 1 2017, 7:15 PM