Page MenuHomePhabricator

Use subrepos instead of git submodules for deployed MediaWiki extensions
Closed, DeclinedPublic

Description

This will merge the extensions into the deployment branch of mediawiki core, rather than branching and cloning 100+ submodules each week. Changes to the extensions would be merged into the relevant deployment branches of mediawiki/core. Branches can be used in the extensions to track deployed changes, but this wouldn't be necessary since the core deployment branch would be tracking the deployed changes.

Advantages:

  1. Branching of extensions could be simplified.
  2. Extension developers / maintainers control the branching and tagging of their code, no auto-generated branches cluttering up the repos.
  3. One view of deployed code - the git history of the deployment branches clearly shows the history of what changed in core as well as the extensions.
  4. Hotfixes to either core or the extensions show up as squashed merges into the deployed branch in one linear history.
  5. Easier patch management for security patches as all patches could be continuously maintained, rebased on top of the deployed branch. Eliminates a significant patch management overhead for release engineering.
    • This also eliminates significant risk of human error in the process.
  6. The process of branching for deployment on Wednesday will be significantly faster: between patch management simplification and faster branching, I expect to save somewhere between 30-60 minutes every week.
  7. swat deployments should be easier as well. Instead of cherry-picking a patch and then updating the submodule pointer in a separate commit, you would just propose the patch to the deployed branch directly and you wouldn't need a separate submodule bump commit.

I'm not entirely sure how to make the transition to the new setup, or how to get people familiarized with it and comfortable with sub-repos. To start with I need to publish my example repo so that interested parties can see what the git history looks like with this arrangement. After that I don't really know what the next step is going to be, suggestions are welcomed.

Related Objects

StatusAssignedTask
OpenNone
OpenNone
StalledNone
Declinedmmodell
Declinedmmodell
InvalidNone
Resolvedmmodell
ResolvedJdforrester-WMF
Declinedmmodell
Resolvedmmodell
Resolvedmmodell
Resolvedmmodell
OpenNone
ResolvedKrinkle
ResolvedKrinkle
Resolvedmmodell
DuplicateKrinkle
ResolvedKrinkle
ResolvedKrinkle
ResolvedMaxSem
ResolvedKrinkle
ResolvedKrinkle
ResolvedKrinkle
ResolvedKrinkle
ResolvedKrinkle

Event Timeline

mmodell created this task.May 12 2015, 8:38 AM
mmodell claimed this task.
mmodell raised the priority of this task from to Normal.
mmodell updated the task description. (Show Details)
mmodell added subscribers: Yurik, fgiunchedi, mobrovac and 11 others.
mmodell added a comment.EditedMay 12 2015, 11:37 AM

One thing I'm a little unsure of: sub-sub-modules - currently 3 of the deployed submodules have their own submodules:

One thing I'm a little unsure of: sub-sub-modules - currently 3 of the deployed submodules have their own submodules:

We don't actually deploy those submodules since the zipball composer gets doesn't include them.

mmodell added a comment.EditedMay 14 2015, 4:37 PM

Oh, interesting.

But that still leaves VisualEditor and Collection... I'll have to either subtree-merge those or keep them as submodules.

Just making sure. Are you referring to git subrepo in this task? If so, as per its spec, sub-sub repos are supported as well (according to its docs, at least).

mmodell added a comment.EditedMay 14 2015, 5:45 PM

@mobrovac: yes that's the git-subrepo I am testing out currently. I didn't see anything about nested subrepos, where did you find it documented?

Also, I'm not too thrilled about the fact that it's written in bash, but it's a joy to use and I really like the general model, it seems to get the best features of submodules without the headaches.

@mobrovac: yes that's the git-subrepo I am testing out currently. I didn't see anything about nested subrepos, where did you find it documented?

From https://github.com/ingydotnet/git-subrepo/#benefits : Subrepo repositories can contain subrepos themselves

Also, I'm not too thrilled about the fact that it's written in bash

True, but at least that allows for quick, painless and no-deps install and set-up, which given that we plan to use it for prod deployments, is a big plus IMHO.

but it's a joy to use and I really like the general model, it seems to get the best features of submodules without the headaches.

Yup, +1 from me as well.

mmodell moved this task from To Triage to In-progress on the Deployments board.Jun 1 2015, 6:00 PM
chasemp added a subscriber: chasemp.Jun 2 2015, 9:46 PM

Following up from a conversation we had in #wikimedia-releng on irc (CC'ing interested-parties):

One thing that I would have to come up with a solution for is how to add the VisualEditor/lib/ve subtree - if visual editor extension used git-subrepo to merge in lib/ve then it wouldn't be a problem. I'm sure I can come up with a different work around if the VE team objects to the idea of using git-subrepo, but that would probably be the easiest way to handle it if we go the route of git-subrepo for release branches.

My experience so far is that git subrepo pull --all is quite slow. I don't think it's any slower than make-wmf-branch but it's not much faster. I certainly would like to see a significant improvement there but I'm not sure there is any solution - there are so many extensions, any way you go about it is bound to be a little slow.

Following up from a conversation we had in #wikimedia-releng on irc (CC'ing interested-parties):
One thing that I would have to come up with a solution for is how to add the VisualEditor/lib/ve subtree - if visual editor extension used git-subrepo to merge in lib/ve then it wouldn't be a problem. I'm sure I can come up with a different work around if the VE team objects to the idea of using git-subrepo, but that would probably be the easiest way to handle it if we go the route of git-subrepo for release branches.

Merging one of the (possibly many) integrations of VisualEditor (VE-MW) into the main stand-alone VE repo, or mastering VisualEditor inside said integration, is a pretty strong "please, no" from our end. Do you have a better solution?

@Jdforrester-WMF: It wouldn't necessarily have to take place in the main VE repo, this could be done via an intermediate merge repo that simply merges the subrepo and then gets merged into mediawiki/core.

This is really a detail specific to the wmf release branches of mediawiki, it will really only come into play when applying commits from the various extensions to the wmf releases.

Also, this may not have been stated clearly: this is an experiment - I'm not demanding any changes from the editing team, it's really just an idea I'm exploring to see if it really solves the problems identified in the task description, and whether it solves more problems than it causes.

Mattflaschen-WMF edited subscribers, added: VisualEditor; removed: Contributors-Team.
mmodell changed the task status from Open to Stalled.Jun 15 2015, 6:05 PM

Either this or something similar will become important when we move mediawiki deployments to use the new deployment system instead of scap. That will probably be next quarter (septemberish?)

In the meantime this is stalled I think.

greg moved this task from INBOX to Backlog on the Release-Engineering-Team board.Sep 24 2015, 6:25 AM
greg moved this task from Backlog to Next on the Release-Engineering-Team board.Sep 28 2015, 11:20 PM
mmodell changed the task status from Stalled to Open.Nov 25 2015, 6:18 PM
mmodell raised the priority of this task from Normal to High.
mmodell lowered the priority of this task from High to Normal.Jan 28 2016, 5:22 PM
Addshore renamed this task from Use subrepos instead of git submodules for deployed MediaWiki extensions to Use subrepos instead of git submodules for deployed MediaWiki extensions.Jul 29 2016, 2:42 PM
mmodell closed this task as Declined.Aug 10 2016, 4:45 PM

I don't think this is in the cards after all.

Seb35 added a subscriber: Seb35.Oct 7 2017, 3:43 PM