With the current option roster, I think I'll go with option 4 which I named the debian-like kung-fu.

I believe that one has all the tradeoff in the right balance, and laves the most critical part (version bump) to a manual git commit.

Both the helm chart and the container image have the same version number, which is otherwise just an incremental number (no special semantics). I have no strong opinions if the version number should contain timestamp and git hash, as long as is incremental with no special semantics.

The artifacts are released at the same time. I believe that in our context, doing this in a different way will just cause more confusion and more moving parts for no added benefit.
This release step can be accompanied by a git tag for extra clarify, but is otherwise not required in the workflow, and should not break anything if not present.

Finally, the shipping part happens at this moment, when the version is bumped.

The developer has the following stages when releasing / deploying to production:

• test locally, possibly using lima-kilo (unrelated to this ticket)
• create a pre-release using the workflow described above, targeted for toolsbeta (example, like 1.2.3~toolsbeta1)
• do adjustments and iterate, if required (continuing with the example, 1.2.3~toolsbeta2).
• then once happy, merge another patch to bump version (and trigger shipping etc) (from 1.2.3~toolsbeta1 to 1.2.3, which sorts higher)

Worth noting that this is the workflow currently at play for most of the debian packages we have, like jobs-framework-cli and others. And it works just fine in my opinion.

In T339198#8935771, @aborrero wrote:

With the current option roster, I think I'll go with option 4 which I named the debian-like kung-fu.

I believe that one has all the tradeoff in the right balance, and laves the most critical part (version bump) to a manual git commit.

Both the helm chart and the container image have the same version number, which is otherwise just an incremental number (no special semantics). I have no strong opinions if the version number should contain timestamp and git hash, as long as is incremental with no special semantics.

Can you elaborate what means for you incremental?

Does it mean that it increases on every build? (ex. it depends on when it's being built)
That it increases on every commit to the repo?
Does it have to be strictly sequential? (no jumps in the numbers)
All of the above?

The artifacts are released at the same time. I believe that in our context, doing this in a different way will just cause more confusion and more moving parts for no added benefit.
This release step can be accompanied by a git tag for extra clarify, but is otherwise not required in the workflow, and should not break anything if not present.

Finally, the shipping part happens at this moment, when the version is bumped.

The developer has the following stages in the before releasing / deploying to production:

• test locally, possibly using lima-kilo (unrelated to this ticket)
• create a pre-release using the workflow described above, targeted for toolsbeta (example, like 1.2.3~toolsbeta1)

Helmfile requires semantic version format, I guess than the equivalent would be 1.2.3-toolsbeta1.

Note that creating this pre-release does not mean that you can deploy it anywhere, the deployment is handled in the toolforge-deploy repo, not here, so the flow with deployment to toolforge would have to be:

• create MR bumping the version on the component repo to 1.2.3-toolsbeta1
  • how is it generated?
  • is it done manually?
• get it reviewed and merged, this pushes that version chart and image
  • to harbor toolsbeta only?
  • to both tools and toolsbeta?
• create MR bumping the version on toolforge-deploy for toolsbeta
• get it reviewed and merged
• manually deploy that change to toolsbeta
• test there, and iterate from the MR creation on the component repo with version 1.2.3-toolforgeN where N is the number of iterations
• once happy, create MR on the component repository to bump the version to 1.2.3
• get it reviewed and merged, this pushes the image and chart to harbor
• create MR bumping the version on toolforge-deploy for toolsbeta
• get it reviewed and merged
• manually deploy that change to toolsbeta
• create MR bumping the version on toolforge-deploy for tools
• get it reviewed and merged
• manually deploy that change to tools
• verify the deployment

Is that correct?
I count at least 5 MRs to do a release, on two different repositories.
This is MRs with 0 code in them, just version bumping (there might be some fixing bugs found, but that is on top of these ones).

• do adjustments and iterate, if required (continuing with the example, 1.2.3~toolsbeta2).
• then once happy, merge another patch to bump version (and trigger shipping etc) (from 1.2.3~toolsbeta1 to 1.2.3, which sorts higher)

Worth noting that this is the workflow currently at play for most of the debian packages we have, like jobs-framework-cli and others. And it works just fine in my opinion.

Personally I don't think that workflow works that well, I always find it cumbersome, error-prone and tedious, but that is a different discussion in a different context.

In T339198#8935849, @dcaro wrote:

Can you elaborate what means for you incremental?

Does it mean that it increases on every build? (ex. it depends on when it's being built)
That it increases on every commit to the repo?
Does it have to be strictly sequential? (no jumps in the numbers)
All of the above?

Just a number that sorts higher than the previous. It increases with each version bump, manually, when the given component is ready for a new release.

Example:

0.0.0
0.0.1
0.0.2
0.0.3
0.1.0
0.1.2
0.1.3

etc.

The numbers don't have any meaning. We could just keep bumping only the last number like 0.0.99999.

I believe semver is losing relevance by the day. On a fun side note:

• create MR bumping the version on the component repo to 1.2.3-toolsbeta1
  • how is it generated?
  • is it done manually?

Yes, manually.

• get it reviewed and merged, this pushes that version chart and image
  • to harbor toolsbeta only?
  • to both tools and toolsbeta?

I don't think we need more than one. We can have everything in the same harbor repo.

As you know, the version to be deployed doesn't depend on where this artifacts are stored. It depends on what helmfiles says for each environment.

Is that correct?
I count at least 5 MRs to do a release, on two different repositories.
This is MRs with 0 code in them, just version bumping (there might be some fixing bugs found, but that is on top of these ones).

Something like this, yes. What you wrote is perhaps the most extreme case, but in a simplified fashion it would be:

• create MR to bump version to be build/released on component repo (it can be the same for toolsbeta/tools)
• create MR to bump version to be deployed on toolforge-deploy repo

If adding the toolsbeta stage then:

• create MR to bump version to be build/released on component repo (it can be the same for toolsbeta/tools)
• create MR to bump version to be deployed on toolforge-deploy repo (toolsbeta)
• create MR to bump version to be deployed on toolforge-deploy repo (tools)

This, by the way, is very similar to the currently well-established workflow with puppet & debian packages:

debian package repository -- same role as -- toolforge component repository
operations puppet tree -- same role as -- toolforge deploy repoistory

You populate a debian repository somewhere (coding, release, etc). Then you make changes in the puppet tree to deploy a given version of the package.

I believe semver is losing relevance by the day. On a fun side note:

I don't think it's losing relevance, but lots of people misuse it yes (us included in a lot of places). It is really really useful when followed though, it both saved a lot of issues and enabled automation in many of the places I have seen continuous delivery working.

The lack of such differentiation between breaking changes and non-breaking ones has always ended up as a blocker for automating the delivery process.

• get it reviewed and merged, this pushes that version chart and image
  • to harbor toolsbeta only?
  • to both tools and toolsbeta?

I don't think we need more than one. We can have everything in the same harbor repo.

This is a big change, I thought that we had decided to have duplicate deployment (tools/toolsbeta) and have those completely independent (that's the reason to have two harbor instances, that was and still is really useful for testing).
If that is not the case, we should discuss it and clarify if we want two independent deployments (as much as possible) or not, probably another decision task?

Is that correct?
I count at least 5 MRs to do a release, on two different repositories.
This is MRs with 0 code in them, just version bumping (there might be some fixing bugs found, but that is on top of these ones).

Something like this, yes. What you wrote is perhaps the most extreme case, but in a simplified fashion it would be:

• create MR to bump version to be build/released on component repo (it can be the same for toolsbeta/tools)
• create MR to bump version to be deployed on toolforge-deploy repo

If adding the toolsbeta stage then:

• create MR to bump version to be build/released on component repo (it can be the same for toolsbeta/tools)

What I understood from your explanation before, is that this would be at least two rounds (one with -toolforge1 and one without).

• create MR to bump version to be deployed on toolforge-deploy repo (toolsbeta)
• create MR to bump version to be deployed on toolforge-deploy repo (tools)

This, by the way, is very similar to the currently well-established workflow with puppet & debian packages:

debian package repository -- same role as -- toolforge component repository
operations puppet tree -- same role as -- toolforge deploy repoistory

You are mssing here the code repository, that would be:
upstream source code repository -- same role as -- toolforge component repository

You populate a debian repository somewhere (coding, release, etc). Then you make changes in the puppet tree to deploy a given version of the package.

Again, debian packages is a very different context, where the person that writes the code is not the same that does the packaging, and it's not the same as the one that installs the package (that's why each of those has their own manual step).
In this case we are the same ones writing the code, building the artifacts, and deploying them, so IMO a single manual step per environment is more than enough and what we should aim for.

In T339198#8938094, @dcaro wrote:

I don't think we need more than one. We can have everything in the same harbor repo.

This is a big change, I thought that we had decided to have duplicate deployment (tools/toolsbeta) and have those completely independent (that's the reason to have two harbor instances, that was and still is really useful for testing).
If that is not the case, we should discuss it and clarify if we want two independent deployments (as much as possible) or not, probably another decision task?

I think it still definitely useful to have a harbor deployment in toolsbeta to, well, test harbor itself as a system.

However, for actual artifact storage and distribution, one might be enough! Happy to discuss this elsewhere if required.

Either having one or two, does not prevent you from storing anywhere you want... in one, in both or doing any other kind of testing or cross-check. And definitely does not invalidate the workflow.

What I understood from your explanation before, is that this would be at least two rounds (one with -toolforge1 and one without).

This is optional, not mandatory. If you want to have some kind of pre-release, this workflow allows for it. I guess that's exactly the point of manual releases.

You are mssing here the code repository, that would be:
upstream source code repository -- same role as -- toolforge component repository

No!

There are debian 'non-native' and 'native' packages (citation). What we do here @WMF is mostly 'native' packages, where there is no separate upstream part.

Examples: jobs-framework-cli, tools-webservice, toolforge-cli, etc.

You populate a debian repository somewhere (coding, release, etc). Then you make changes in the puppet tree to deploy a given version of the package.

Again, debian packages is a very different context, where the person that writes the code is not the same that does the packaging, and it's not the same as the one that installs the package (that's why each of those has their own manual step).

See above about native packages.

Thanks a lot @dcaro for the write-up with all the options. I think it clearly highlights there are multiple axes and a lot of hidden complexity.

My preference is for option 3 (and especially for shipping option 3, on every commit to the main branch).

I would like to understand if we could push it even further, and ship image+chart on each merge request, before it gets merged. My ideal workflow would be that when I open a MR with a change, the CI automatically builds a new image and chart with the changes from my branch, it pushes both to Harbor (either tools or toolsbeta) and I can (optionally) deploy this built chart for testing, either locally on my laptop, or in Toolsbeta, or other test environments (that we don't have yet, but we might build in the future using Magnum, etc.)

I don' have a strong opinion on which versioning scheme we should use, A.B.C-TTTTT-HHHHH seems reasonable because it would avoid conflicts in the scenario I described above where potentially two merge requests are open at the same time, and A.B.C might be the same for both of them. We could even keep A.B.C=1.0.0 forever (or update it very rarely).

In T339198#8939020, @fnegri wrote:

Thanks a lot @dcaro for the write-up with all the options. I think it clearly highlights there are multiple axes and a lot of hidden complexity.

My preference is for option 3 (and especially for shipping option 3, on every commit to the main branch).

I would like to understand if we could push it even further, and ship image+chart on each merge request, before it gets merged. My ideal workflow would be that when I open a MR with a change, the CI automatically builds a new image and chart with the changes from my branch, it pushes both to Harbor (either tools or toolsbeta) and I can (optionally) deploy this built chart for testing, either locally on my laptop, or in Toolsbeta, or other test environments (that we don't have yet, but we might build in the future using Magnum, etc.)

I agree, ideally the same artifact that you tested should be the one that you promote (no rebuilding), we can try to implement something like that, pulling and pushing the image with a new tag/to a different project/namespace/harbor.
It will require having into account that a branch might not hang from latest master, so probably will have to rebase before building and such (and fail if there's merge conflicts, etc.).
I think it's doable.

I don' have a strong opinion on which versioning scheme we should use, A.B.C-TTTTT-HHHHH seems reasonable because it would avoid conflicts in the scenario I described above where potentially two merge requests are open at the same time, and A.B.C might be the same for both of them. We could even keep A.B.C=1.0.0 forever (or update it very rarely).

In T339198#8939020, @fnegri wrote:

Thanks a lot @dcaro for the write-up with all the options. I think it clearly highlights there are multiple axes and a lot of hidden complexity.

My preference is for option 3 (and especially for shipping option 3, on every commit to the main branch).

I would like to understand if we could push it even further, and ship image+chart on each merge request, before it gets merged. My ideal workflow would be that when I open a MR with a change, the CI automatically builds a new image and chart with the changes from my branch, it pushes both to Harbor (either tools or toolsbeta) and I can (optionally) deploy this built chart for testing, either locally on my laptop, or in Toolsbeta, or other test environments (that we don't have yet, but we might build in the future using Magnum, etc.)

What about mixing the two (shipping option 3 and option 4).

Let's call it option 5:. Auto-generating chart & image on every merge request & merge to main (for easy testing) plus having a 'manual' released one every now and then, bumping the version.

The per-MR artifacts would use a version based on the latest "manually" bumped version, examples:

• 1.0.0 <-- manually created version
• 1.0.0+mr1-ttttt-hhhh <-- mr1
• 1.0.0+mr2-ttttt-hhhh <-- mr2
• 1.0.1 <--- manually created version
• 1.0.1+mr3-ttttt-hhhh <-- mr3

The logic to generate the artifacts should not care what the base version is, it will just append "on the fly" the extra string to make the version unique enough.

In T339198#8945900, @aborrero wrote:

In T339198#8939020, @fnegri wrote:

Thanks a lot @dcaro for the write-up with all the options. I think it clearly highlights there are multiple axes and a lot of hidden complexity.

My preference is for option 3 (and especially for shipping option 3, on every commit to the main branch).

I would like to understand if we could push it even further, and ship image+chart on each merge request, before it gets merged. My ideal workflow would be that when I open a MR with a change, the CI automatically builds a new image and chart with the changes from my branch, it pushes both to Harbor (either tools or toolsbeta) and I can (optionally) deploy this built chart for testing, either locally on my laptop, or in Toolsbeta, or other test environments (that we don't have yet, but we might build in the future using Magnum, etc.)

What about mixing the two (shipping option 3 and option 4).

Let's call it option 5:. Auto-generating chart & image on every merge request & merge to main (for easy testing) plus having a 'manual' released one every now and then, bumping the version.

The per-MR artifacts would use a version based on the latest "manually" bumped version, examples:

• 1.0.0 <-- manually created version
• 1.0.0+mr1-ttttt-hhhh <-- mr1
• 1.0.0+mr2-ttttt-hhhh <-- mr2
• 1.0.1 <--- manually created version
• 1.0.1+mr3-ttttt-hhhh <-- mr3

The logic to generate the artifacts should not care what the base version is, it will just append "on the fly" the extra string to make the version unique enough.

I think that the benefit comes only when the generated artifact is ready to use (ex. to be bumped on the toolforge-deploy repo).
In that sense, anything that does not publish a production ready artifact on merge for me lacks the core benefit of continuous delivery (and thus, is a very similar solution to just manually building and pushing).

Having to do 2 manual actions (or more) to get the code deployed in an environment is one manual action too much xd

I'd go with option 3, with slightly preference over other possible options still using shipping option 3 (continuous delivery), but strong preference to other non-shipping option 3 solutions.

It's been a week since this, I set the deadline to either setup a meeting or get consensus for the 29th (one more week).
Please share your opinions! Read and consider all of the others too!

I just added option 5:

• Version option: 1 (always incremental, no special semantics)
• Artifacts option: 2 (chart and container image at the same time)
• Shipping option: 3 (ship on every merge/push to main)

So my vote goes to either option 4 or option 5.

I'm equally fine with option 3 or option 5.

In both cases, I'm not sure I see any benefit in incrementing the version (X in 0.0.X-TTTTT-HHHHH or C in A.B.C-TTTTT-HHHHH), instead of keeping the first part of the version string constant (0.0.0-TTTTT-HHHHH or something similar). Ideally I would use simply TTTTT-HHHHH but I think Helm does not allow that.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/11

toolforge-cd: adapt to the decided flows

dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/11

toolforge-cd: adapt to the decided flows