@Eileenmcnaughton guessing this still needs doin!

I've been out a few days but will try and catch up with you & @hashar to see if we can push it over the line.

Change 685528 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/dev@master] Use upstream civicrm-buildkit

https://gerrit.wikimedia.org/r/685528

Change 688350 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[integration/config@master] Update wikimedia-fundraising-civicrm-docker to pull civicrm-buildkit from upstream repo.

https://gerrit.wikimedia.org/r/688350

@hashar I've pushed up the update to the CI job and will hopefully catch up with you tomorrow on IRC to see if it's +2'able so we can try it out. Thanks in advance!

Change 689947 had a related patch set uploaded (by Jforrester; author: Jgleeson):

[integration/config@master] jjb: [wikimedia-fundraising-civicrm-docker] Use upstream civicrm-buildkit

https://gerrit.wikimedia.org/r/689947

Change 688350 merged by jenkins-bot:

[integration/config@master] dockerfiles: [civicrm] Don't provide the Wikimedia fork of civicrm-buildkit

https://gerrit.wikimedia.org/r/688350

Mentioned in SAL (#wikimedia-releng) [2021-05-12T16:15:22Z] <James_F> Docker: Publishing civicrm 0.2.1 for T277500

Change 685528 merged by AndyRussG:

[wikimedia/fundraising/dev@master] Use upstream civicrm-buildkit

https://gerrit.wikimedia.org/r/685528

Change 692107 had a related patch set uploaded (by AndyRussG; author: AndyRussG):

[wikimedia/fundraising/dev@master] Get buildkit from github with anonymous https

https://gerrit.wikimedia.org/r/692107

Change 692107 merged by Eileen:

[wikimedia/fundraising/dev@master] Get buildkit from github with anonymous https

https://gerrit.wikimedia.org/r/692107

Is there more in review here?

Change 692693 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/crm@master] Update CI build files to use new civicrm-buildkit

https://gerrit.wikimedia.org/r/692693

On the buildkit issue I synced our repo today - there was one patch we hadn't upstreamed & a bit of mess but I fixed that up and then did a commit like

git pull upstream master -s recursive -X theirs --squash

And merged that

It's not possible to do merge commits as far as I can see as they get rejected on there being too many commits out of sync so only ever committing upstream & doing ^^ manually every week is the best we have at the moment. Doing the above DID fix the immediate bug.

The civicrm repo symlink part of this is getting lost - maybe it needs a separate phab as it is equally high priority

Houston we have a problem

@Eileenmcnaughton merge squash is one way to sync upstream but you loose all the upstream git history and that might prove to be a bit complicated when porting patches.

I suspect the issue you ran into is that locally you have:

• updated the upstream branch
• crafted a merge commit
• tried to send that to our Gerrit

The topology on your local machine would be something like:

* (master) merge upstream master
| \
| * (upstream/master) Latest change at upstream
| * <LOT OF COMMITS>
| * <LOT OF COMMITS>
* | (origin/master) Wikimedia specific patch
| /
* Common point from the past

Sending your merge commit also sends the ''lot of commits'' made by upstream and you would run in several problem:

• if using git-review, since the commits are not known it will warn about attempting to send more than X commits for review
• our Gerrit is configured to NOT allow more than 10 commits to be send to review and thus that get rejected
• upstream commits are made by other authors than you and would be rejected unless the repository/branch is configured to allow your account to Forge Author Identity
• upstream commits do not have the Change-Id header in the commit messages which used by Gerrit to associate a commit with a change

And indeed from your comment at https://gerrit.wikimedia.org/r/c/integration/config/+/689947/ :

I tried to follow it up with a merge commit to convince gerrit it was up to date - but I failed

Do you really want to submit the above commits?
Type 'yes' to confirm, other to cancel: yes
remote:
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done
remote: commit 269d98a: warning: subject >100 characters; use shorter first paragraph
remote: commit 269d98a: warning: too many message lines longer than 120 characters; manually wrap lines
To ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm/civicrm-buildkit.git
! [remote rejected] HEAD -> refs/for/master%topic=wmf (the number of pushed changes in a batch exceeds the max limit 10)
error: failed to push some refs to 'ssh://eileen@gerrit.wikimedia.org:29418/wikimedia/fundraising/crm/civicrm-buildkit.git'

Sending the merge commit with all commits made to upstream is rejected. The limitation has been set exactly to catch that.

How to do it

I thought I wrote somewhere how to address the use case of syncing an upstream repository maybe that was just on IRC. The generic idea is to manually sync the upstream repository by pushing their commit and tags directly to the git repository without using the review mechanism.

Assuming your local repository has remotes:

We would create branches in our Gerrit repository to allow direct push. Assuming the branch is named upstream one would then be able to do:

git remote update upstream
git push origin --tags upstream/master:upstream

Which does not result in any change to be created nor any complain being made because of the 10 changes limit. At that point Gerrit now knows about all the upstream commits.

Then on your local machine the latest change from upstream is known to gerrit (origin/uptream):

* (master) merge upstream master
| \
| * (upstream/master) (origin/upstream) Latest change at upstream   # <----- known to Gerrit
| * <LOT OF COMMITS>
| * <LOT OF COMMITS>
* | (origin/master) Wikimedia specific patch
| /
* Common point from the past

When you send for review the unsquashed merge upstream master merge commit, Gerrit knows all the commits from both parents and a single change will be crafted. You retain the whole history from upstream.

To browse the repository later on and find your custom patches you can git log filtering out the upstream commits (using --first-parent) and merge commits (--no-merges). If I got it right:

$ git log --first-parent --no-merges
* Wikimedia specific patch
* Common point from the past

We use that system for operations/software/gerrit which is a mirror of the upstream repository. The upstream branches are pushed to our Gerrit with the same name (master > master, and their release branch stable-3.2 is pushed as stable-3.2 as well) and we add our patches to a branch prefixed with wmf/:

I would then do something such as:

git checkout stable-3.2
git pull --tags
git push origin stable-3.2:stable-3.2

git checkout wmf/stable-3.2
git merge stable-3.2   # Update our fork
git push origin HEAD:refs/for/wmf/stable-3.2

Summary

The repository branches are listed at https://gerrit.wikimedia.org/r/admin/repos/wikimedia/fundraising/crm,branches and seems to be the upstream one. If needed, you might grant the Push permission for refs/heads/* and refs/tags/* at https://gerrit.wikimedia.org/r/admin/repos/wikimedia/fundraising/crm,access

Then pick a branch name that would be used to maintain our custom patch. It could be deployment or wmfstaging or whatever really. That branch should probably by the default (in the branch view edit HEAD to point to wmfstaging). Optionally add a permission that prevent direct pushes to that branche forcing people to send changes for reviews.

And hopefully you are set.

The set of commands I am using to sync Gerrit source code and manage our patches is described at https://wikitech.wikimedia.org/wiki/Gerrit/Upgrade#Upgrade_Gerrit_core_branch , though that uses tags not branches.

@Ejegg @jgleeson can you take a look at the above where @hashar has spelt out how we could do this.

Currently we have 2 repos.

gerrit ssh://eileen@gerrit.wikimedia.org:29418/wikimedia/fundraising/crm/civicrm-buildkit.git (fetch)
gerrit ssh://eileen@gerrit.wikimedia.org:29418/wikimedia/fundraising/crm/civicrm-buildkit.git (push)
upstream https://github.com/civicrm/civicrm-buildkit.git (fetch)
upstream https://github.com/civicrm/civicrm-buildkit.git (push)

If I do

eileen@eileen-laptop:~/dev/fundraising-dev/src/civicrm-buildkit$ git diff upstream/master gerrit/master

then the only difference is the addition of the .gitreview file - so we are where we want to be file wise.

Over the past 6 years there really was only one legitimate reason for us to ever be out of sync with the upstream (the composer 1 vs 2 issue - although I think we could deal with a similar issue without needing to fork in future with the tools we have now).

So my feeling is that the goal is really about the files being in sync in order to keep jenkins running 'as if we hadn't forked'. Hence it might be a feature not a bug if our commit log is fairly meaningless and people have to look at the commit log on the upstream repo

However, if we look at the above it also offers the possibility? of having a branch that we push to that ?? maybe we can use directly? which is just a sync?

@hashar - thank you for the great write up

Glad you like it! I wrote it while my family was preparing a surprise brunch and I felt like you would enjoy having a wall of text this morning rather than tomorrow or even later. Timezones are complicated.

@hashar - it even has big bold headers :-)

I hope you liked the brunch. Sounds like you may not have been surprised

CiviCRM tests are currently failing on CI due to having one of these patches (https://gerrit.wikimedia.org/r/c/integration/config/+/688350/4 ) merged and not the other (https://gerrit.wikimedia.org/r/c/integration/config/+/689947 ).

Example failure: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/6071/console

I'm strongly in favor of getting rid of our fork and directly using upstream.

Newer versions of Buildkit are compatible with older versions of Civi - buildkit is frequently used by CiviCRM professionals to spin up environments to debug a client's problem (when the client may be months or even years behind). The latest version of CiviCRM is NOT necessarily compatible with older versions of buildkit, though - so
we're much more likely to see failures because we've forgotten to update our buildkit fork than we are likely to see failures due to an unexpected upstream update.

@Eileenmcnaughton has maintainer rights on the upstream, so we can easily fix problems that arise there.

I have made some suggestions on https://gerrit.wikimedia.org/r/c/integration/config/+/689947/1/jjb/wm-fundraising.yaml . Then to grab civicrm-buildkit directly from upstream there are a few solutions:

Composer

The composer.json already has a couple repositories:

"repositories": [
  {
    "type": "git",
    "url": "https://github.com/ejegg/login-and-pay-with-amazon-sdk-php"
  },
  {
    "type": "git",
    "url": "https://github.com/eileenmcnaughton/omnimail-silverpop"
  }
],

The exact reference / tag to install being in the composer.lock as I understand it?

bin/ci-populate-dbs.sh

The script already set some environment variable for the buildkit:

CIVICRM_BUILDKIT="${WORKSPACE}/src/wikimedia/fundraising/civicrm-buildkit"
if [ -d /src/wikimedia/fundraising/crm/civicrm-buildkit ]; then
  # For CI Docker container
  CIVICRM_BUILDKIT=/src/wikimedia/fundraising/crm/civicrm-buildkit
fi

Maybe it can be used to git clone / checkout a specific reference. That sounds a bit hacky though.

Then drop our Gerrit mirror.

so potentially we could do this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/724358

And if we want to diverge from master at any point we can specify the tag

(our local dev builds use master so unless something started failing staying in sync is best)

Change 724215 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Revert \"dockerfiles: [civicrm] Don't provide the Wikimedia fork of civicrm-buildkit\"

https://gerrit.wikimedia.org/r/724215

Change 724215 merged by jenkins-bot:

[integration/config@master] Revert \"dockerfiles: [civicrm] Don't provide the Wikimedia fork of civicrm-buildkit\"

https://gerrit.wikimedia.org/r/724215

Mentioned in SAL (#wikimedia-releng) [2021-09-29T15:52:38Z] <hashar> Successfully published image docker-registry.discovery.wmnet/releng/civicrm:0.2.2 # T277500

Change 724772 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: use proper path for Civicrm buildkit

https://gerrit.wikimedia.org/r/724772

I have deployed @jgleeson patch which set PATH to the civicrm-buildkit cloned repository. That should fix the build until it is found how to replace the git clone CI is doing by something else.

Change 724772 merged by jenkins-bot:

[integration/config@master] jjb: use proper path for Civicrm buildkit

https://gerrit.wikimedia.org/r/724772

@hashar we just merged a patch to checkout buildkit if not present. It should be the case that you can stop checking out the buildkit repo now & it will continue to work (& that if we want to tag to a commit in future we can do that without involving you as the config will be in a repo we have rights to).

Change 692693 abandoned by Jgleeson:

[wikimedia/fundraising/crm@master] Update CI build files to use new civicrm-buildkit

Reason:

no longer needed

https://gerrit.wikimedia.org/r/692693

Change 732064 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[integration/config@master] Remove zuul clone of civicrm buildkit as this now happens within the project ci build file bin/ci-populate-dbs.sh

https://gerrit.wikimedia.org/r/732064

Change 689947 abandoned by Jforrester:

[integration/config@master] jjb: [wikimedia-fundraising-civicrm-docker] Use upstream civicrm-buildkit

Reason:

https://gerrit.wikimedia.org/r/689947

Change 732086 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[integration/config@master] Remove zuul clone of civicrm as we're gonna check the code in directly to the parent 'crm' project, no longer needing the existing submodule approach, which refers to the zuul cloned repo.

https://gerrit.wikimedia.org/r/732086

Change 732064 merged by jenkins-bot:

[integration/config@master] Remove zuul clone of civicrm buildkit

https://gerrit.wikimedia.org/r/732064

Change 732086 abandoned by Jgleeson:

[integration/config@master] Remove zuul clone of civicrm

Reason:

https://gerrit.wikimedia.org/r/732086