Quoted Text

The Problems:

1. CLIs deployment problem:

There is currently no way in the deploy cookbook to deploy a cli package (usually done using a cli repo release branch), and execute new/modified functional tests from a new toolforge-deploy branch, unless you give the release branch on the cli repo, and the new/modified functional tests branch on toolforge-deploy the same name (This is not exactly realistic).

This is because the cli is deployed from the release branch on the cli repo,
while the functional tests that are run after each deployment lives in the toolforge-deploy repo (This is likely structured like this because the cli deb package is gotten from the release patch on the cli repo and uploaded to our internal deb repositories, during the cookbook run).

This is also applicable (in a way) to the APIs deployment using the deploy cookbook. The difference is that since the APIs release branch and the functional tests are in the same repo, you can checkout the release branch (created by the cicd pipeline) locally, add your new/modified tests, then update the release branch with this new change.

2. Problem running the gitlab tests and deployment cicd pipelines for PRs from external contributors

more context: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/70#note_136315
TL;DR is that deployment pipelines doesn't work for PRs from non-maintainers.

...(Add more problems here)...

Outline of the current deployment flow for the API:

For APIs, the flow goes like this: (Please indicate if I missed something)

• submit a PR, and the cicd pipeline builds the dev container image and helm chart and upload them to toolsbeta harbor, these will be used for testing this PR on lima-kilo.
• once the PR has been tested, approved and merged, cicd pipeline does the following:
  • builds the toolsbeta and tools container image and helm chart and upload these to toolsbeta and tools respectively.
  • creates a gitlab release by tagging the merged git commit
  • creates a toolforge-deploy upgrade branch. This branch will be used to deploy the api on toolsbeta and tools k8s clusters using the deploy cookbook, and the deployment logs and functional tests will be logged in this PR. This PR will be merged after the deployments has been confirmed as successful.

Outline for the current deployment flow for the Clis:

For the CLIs, the flow goes like this: (Please indicate if I missed something)

• submit a PR, and the cicd pipeline builds the .deb package, zips it, and saves it as a build archive on gitlab. This zip file is downloaded, unzipped and installed in lima-kilo by toolforge_deploy_mr.py script for local testing. (This script currently lives in lima-kilo repo, but should probably be moved to toolforge-deploy repo. Making a mental note of that so I don't forget).
• once the PR has been tested, approved and merged, the cicd pipeline builds another .deb package, zips and saves it in gitlab as a build artifact.
• After merging you manually do the follow:
  • fetch the merge locally and checkout a bump_version branch where you run the ./utils/bump_version.sh script on your local machine. This script currently exists on all our clis and is responsible for creating release branch and tag. Then you manually upload this release branch and tag.
  • After creating the release branch on the cli repo, the cicd pipeline builds the package, but this time uploads it to pypi.
  • Next step is to run deploy cookbook this install this package on the bastions and runs the functional tests, but unlike the APIs, the installation and functional tests logs are logged to the cli release PR, not toolforge-deploy (since the release PR is created on the cli repo and not toolforge deploy). Once the installation and functional tests has been confirmed as successful, we merge the cli release PR, then manually create a gitlab release for this. (You can already see that the deployment flow for the clis is more hairy than that of the APIs)

Proposed Fixes:

This will probably be in multiple stages:

• replace --git-branch option in the deploy cookbook with --deploy-branch and --test-branch. This will allow us to independently choose the deployment branch, and the test branch, and will fix the current issue. If --test-branch is not provided, it can default to the value of --deploy-branch.
• Simplify the deployment flow for the CLIs (maybe become more like the flow for the APIs) (more on this later...)
• Work on deploying patches created by non-maintainers, taking care that secrets are handled properly and doesn't leak (more on this later...)

...