Debian package README
AbandonedPublic

Authored by thcipriani on Dec 19 2017, 6:00 PM.

Details

Reviewers
mmodell
demon
Group Reviewers
Release-Engineering-Team
Patch without arc
git checkout -b D923 && curl -L https://phabricator.wikimedia.org/D923?download=true | git apply
Summary

For the master branch, the information in the README should just be a
warning about cutting from this branch for a production release. For the
release branch, this README will be more involved.

Diff Detail

Repository
rMSCA Scap
Branch
master
Lint
No Linters Available
Unit
Unit Tests OK
Build Status
Buildable 2613
Build 4344: docker-diffsJenkins
Build 4343: differential-jessieJenkins
Build 4342: arc lint + arc unit
thcipriani created this revision.Dec 19 2017, 6:00 PM
Restricted Application added a reviewer: Release-Engineering-Team. · View Herald TranscriptDec 19 2017, 6:00 PM
Restricted Application added a project: Release-Engineering-Team. · View Herald Transcript
thcipriani requested review of this revision.Dec 19 2017, 6:02 PM
demon added a comment.Dec 19 2017, 7:38 PM

Won't this create a merge conflict each time?

In D923#18438, @demon wrote:

Won't this create a merge conflict each time?

If we merge and fix the conflict once by overwriting the file with the other version, then no it shouldn't conflict after that.

In D923#18450, @mmodell wrote:
In D923#18438, @demon wrote:

Won't this create a merge conflict each time?

If we merge and fix the conflict once by overwriting the file with the other version, then no it shouldn't conflict after that.

Plus it should be an easy conflict fix: git checkout --{ours,theirs} (whichever one doesn't have the flying pig :)). For better or worse the debian directory is full of files with the same name and different contents. I'm not entirely familiar with the master build process for beta, so I don't know what the other options are. Removing the debian directory in master would, I think, be preferable if possible.

hashar added a subscriber: hashar.Dec 19 2017, 11:41 PM

If ./debian is of no use in the master branch maybe it can be removed entirely? This way the master branch is solely code and the release work is being done solely in the release branch.

Another way is switch to a Debian native package which has the source code and debian material all at the same place. That dramatically simplify the workflow since a single commit can change code and the deb package!

I like the debian native package, however, opsen have argued convincingly that it's not the "right" way to do it.

I could probably modify the packaging scripts in CI to do a merge into the release branch (locally in jenkins) and then build packages there based on commits in master.

With a native debian package there will be a single branch master containing the ./debian directory which dramatically simplify the build process. The drawkbacks are:

  • master needs to be always deployable
  • we cant hotfix anymore by cherry picking a patch
  • ops don't like it (there must be good reasons)

So I would recommend to drop the ./debian directory from the master branch. Then for CI:

  • on master branch only run the tests / tox etc
  • on release branch solely run the packaging step, which potentially could use autopkgtest to run the tests Kunal has an example for mediawiki/debian at https://gerrit.wikimedia.org/r/#/c/393435/

The advantage: each branches can evolves at their own speed:

  • master can have new features / potentially breaking changes, it is not going to impact prod until we cut a new release
  • release (and hence the deployed package) can receives hotfixes cherry picked from master. Probably via debian/patches

The drawback:

Cutting a release requires a bit more time since one has to catch up with all the changes made in master. But potentially each such change could also come with a patch to the release branch.

Eg if one has a new python module requirement, the flow would be roughly:

  • do the needful in master. CI runs tox / tests etc
  • once merged, propose a patch in release that merge the change and bump the requirement in Depends:. CI build the package and runs the autotest using the installed packages.
demon added a comment.Dec 20 2017, 6:49 PM

I'm in favor of dropping the debian/ directory from master. Then there's zero confusion. The package can still be installed from master using distutils/setuptools. (Cf: D929: WIP: Dropping bin/scap in favor of setup.py support as well)

In D923#18545, @hashar wrote:

So I would recommend to drop the ./debian directory from the master branch. Then for CI:

  • on master branch only run the tests / tox etc
  • on release branch solely run the packaging step, which potentially could use autopkgtest to run the tests Kunal has an example for mediawiki/debian at https://gerrit.wikimedia.org/r/#/c/393435/

The problem with this is that we currently build a package for every commit to master and deploy that to beta cluster. This is an important step prior to cutting a release and I don't want to eliminate that. So I'll have to revise the jenkins job to also merge to release before building the package (or copy the debian/ stuff from release to a new branch based on master, or some other thing like that)

In D923#18569, @mmodell wrote:

The problem with this is that we currently build a package for every commit to master and deploy that to beta cluster. This is an important step prior to cutting a release and I don't want to eliminate that. So I'll have to revise the jenkins job to also merge to release before building the package (or copy the debian/ stuff from release to a new branch based on master, or some other thing like that)

Yup, you can have the job:

  • checkout release
  • attempt to merge master
  • build a snapshot of the result
  • deploy or whine LOUDLY that the package is broken

So this way we continuously build the deb package and can adjust the release branch in case a breaking change to master did not come with a counterpart in release.

thcipriani abandoned this revision.Jan 2 2018, 4:35 PM
In D923#18655, @hashar wrote:
  • checkout release
  • attempt to merge master
  • build a snapshot of the result
  • deploy or whine LOUDLY that the package is broken

    So this way we continuously build the deb package and can adjust the release branch in case a breaking change to master did not come with a counterpart in release.

Nice. Let's try to do this :)