Page MenuHomePhabricator

Create new civicrm repo
Closed, ResolvedPublic2 Estimated Story Points


We want to change the civicrm repo structure significantly, plus base our contrib branch on upstream. We probably want a fresh repo to do this in.

Change deployment scripts if necessary.

Ensure that the recursive submodules work correctly.

Current status:
civi-4.6.9: The finished development master for civi 4.6.9 with WMF patches

  • master + civi-upstream-4.6.9 + patches to customize civi + patches to fix WMF modules when running on 4.6.9

civi-4.6.9-deployment: Deployment version of civi-4.6.9, but done as a shortcut rather than a true merge to the "deployment" branch. Dead-end. Also, note that this branch will not be updated beyond the version used to do the schema migration.

  • civi-4.6.9 + hacky deployment patch

master-plus-4.6.9: master + civi-4.6.9, this will become the new master.
deployment-prep-4.6.9: deployment + master-plus-4.6.9, this will become the new deployment

Event Timeline

awight created this task.Oct 6 2015, 11:03 PM
awight raised the priority of this task from to High.
awight updated the task description. (Show Details)
awight added subscribers: Aklapper, atgo, awight and 2 others.

Moving to "not fr-tech" column because I think this is a Jeff thing. Please correct my mistake if I'm wrong :)

Eileenmcnaughton added a comment.EditedOct 9 2015, 3:31 AM

This is probably something we need to co-ordinate with Jeff.

The way the wmf CiviCRM repo currently works is that the civicrm deployment tarball is checked into the repo and patches are applied / re-applied on top.

The way the CiviCRM repo works is that there are 3 repos ++(and a little bit of '--')



  • civicrm
    • packages
    • drupal

The civicrm-core repo has the information to build some of the files (the DAOs) but not the actual files. These are generated by a script which also runs composer & bower_install and pulls in the packages & drupal dirs.

The same script deletes the test folder, the tools folder, the xml folder and a handful of other files

Creating core-style branches
The advantage of having repos set up like the core repo is that we can more easily cherry-pick patches & prepare them for upstream. Currently we have a patches that cross the drupal repo & civicrm repo and when we cherry-pick back we have to deal with conflicts on folders like 'tests' which aren't in our repo. In generally cleaner patching feels safer.

My preference is to create a branch per release and when a new release comes our rebase our patches over it e.g

git checkout civi-4.6.9
git checkout -b civi-4.6.10
git pull --rebase upstream 4.6.10
git review

Note that in the above CiviCRM has tags rather than branches for point versions & git review would need to be edited. Generally each time we change point version we can drop some patches that have gone into core in the meantime or we may need to edit some.

The Fuzion repo

Fuzion currently has it's own repo with patches and the code wmf is running on staging is actually patches over the top of the Fuzion repo. In general the Fuzion repo is the current Civi version + patches that are slated for a future release rather than Fuzion-specific things. In the past it has been the basis of the LTS but it's not carrying the volume of patches this time that it did in 4.4 & 4.2.

Currently in the Fuzion repo we have reverted a search 'enhancement' which caused performance issues which was a key reason I went for the Fuzion repo as the base. Probably when we switch to the new repo structure we will take the patch set at that time & not keep any sync with the Fuzion repo other than cherry-picks

This is the current wmf core repo I'm used to generate the code in the civi-4.6.9 branch
and the current Fuzion repo

Proposed repos

  • civicrm-core
  • civicrm-drupal
  • civicrm-packages
  • civicrm

Proposed process
We would need to put the first 3 under git review
We would need to set up a way to run the packaging script. The way this currently runs on Fuzion is that web have used buildkit to create a 'dist' site - the civibuild script runs nightly and generates tarballs off our main fuzion version & the wmf branch - ie.

cd /srv/www/buildkit/dist && env GIT_REMOTE=fuzion cividist update && env FILE_SUFFIX=nightly cividist build fuzion/4.6

We could go with a similar process but the cron wrapper would probably go a step further & untar the tarball and check it into the repo/branch that holds the deployable code, preferably with some message that says which commits it includes but that might be stage 2.

The deployment script can be run without the civibuild dist site - it's just harder.

With buildkit installed the code to setup the dist site is civibuild create dist

If we are using the buildkit tools we don't want to also have to generate the Joomla! & WP tarballs if we can help it - I logged a note to buildkit about that

DStrine set Security to None.Oct 13 2015, 5:28 PM
DStrine edited a custom field.

@Eileenmcnaughton: What do you think about doing the initial upgrade using our old civi-4.6.9 branch, then migrating to the new repo structure at a later date (in the immediate future)? That would reduce the complexity of both the upgrade, and of the repo migration, which would be a near no-op...

Yes - I agree - I was imagining we would go live with the code we have been testing on staging. I did do a commit for additional minor changes that came from the repo re-configure - but I think I noted in gerrit that I thought it made more sense to not include that in the upgrade since it's not what has been tested

awight updated the task description. (Show Details)Nov 2 2015, 6:50 PM
Eileenmcnaughton closed this task as Resolved.Nov 10 2015, 5:42 PM
mmodell removed a subscriber: awight.Jun 22 2017, 9:36 PM
Jgreen moved this task from Triage to Done on the fundraising-tech-ops board.Feb 13 2020, 7:40 PM