Page MenuHomePhabricator

Automate the wbSuite Release Process
Closed, ResolvedPublic5 Estimated Story PointsSpike

Description

Goal: Analyse the release process and identify steps that can be automated.

Context: the current release process is fully manual and cumbersome (see T332786). That makes is too time consuming and prone to errors and overlooking things. Threshold for new engineers to step it is also too high. Confidence that we are doing it right is usually low.

Open Question:

  • Check for the "latest" tag. What should we tag as the "latest" release?

Acceptance Criteria:

  • Write tickets to automate the release process

Some tickets may hypothetically be:

  • script for the creation of the needed env files with the needed variables
  • script that checks the last commit hashes of the different extensions, till we get to the point that everything is done for us and we just need to do the last checking 💥
  • store the commit hashes to ensure reproducibility of the releases, see also T340226
  • create make target for dockerhub publish
  • fix the dockerhub publish script https://github.com/wmde/wikibase-release-pipeline/pull/499 (or move all this stuff to CI)
  • make the tarball upload script work with yubikeys and gpg-agent in ssh-mode (or move all this stuff to CI)
  • make it possible to publish images and tarballs no matter whether you are using yubikey or ssh keys (at the moment it's only possible with ssh keys from your .ssh file)
  • build publish workflow in gh actions (CI) - using secrets for keys

Event Timeline

RickiJay-WMDE updated the task description. (Show Details)
RickiJay-WMDE set the point value for this task to 5.
Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptOct 19 2023, 1:41 PM
roti_WMDE changed the task status from Open to In Progress.Nov 20 2023, 9:30 AM
roti_WMDE claimed this task.

I am suggesting the following optimizations:

  • Test Release prep builds on GHCR With this PR main, mw- release branches and PRs merging to releases branches (that is release preparations) publish their docker builds on GHCR (which we could consider being our staging container registry for now). This way, we can easily test release candidates by just pointing our docker-compose files to GHCR.
  • Review Docker Hub Upload Action We have an action to automatically upload Github CI artifacts directly to Docker Hub. It one needs to be tested, potentially fixed and could become our default way to uploading docker releases. This way we would store the docker hub API key in Github Action Secrets (it is there already) and individual devs do not need to bother.
  • Leave tarball upload as a manual process Uploading tarballs currently relies on moving SSH keys around. This is incompatible with Yubikeys. This should probably also not be moved to Github Actions as we probably are not allowed to store production keys on Github. My suggestion is to leave the process as it is for now until we spoke about the future of tarballs. If we really keep them, we will figure out a way to automatically publish them as well.
  • Docs Update Release Checklist Phabricator Ticket Template and this document
  • Reproducible Builds Focus on T340226 soon.
  • Update upstream components [Spike] Find out how to update the versions of locked upstream components once our builds are reproducible.

Does the proposed task: "script that checks the last commit hashes of the different extensions, till we get to the point that everything is done for us and we just need to do the last checking 💥" require " store the commit hashes to ensure reproducibility of the releases, see also T340226"? If so, and its clear something we want/need to do, as part of completing this spike lets make a ticket for it as a subtask of T340226.

The intention of my comment was to replace the task list above. Relevant for the topic you mention is

Update upstream components [Spike] Find out how to update the versions of locked upstream components once our builds are reproducible.

I find those steps a great improvement to the build process. Thanks @roti_WMDE for the summary and the explanations behind them.

I have one question though: are we getting into automating the first steps of the process: changing the .env files and such? or that is work to be done under T349803?

For transparency:

I am suggesting the following optimizations:

  • Test Release prep builds on GHCR With this PR main, mw- release branches and PRs merging to releases branches (that is release preparations) publish their docker builds on GHCR (which we could consider being our staging container registry for now). This way, we can easily test release candidates by just pointing our docker-compose files to GHCR.
  • Review Docker Hub Upload Action We have an action to automatically upload Github CI artifacts directly to Docker Hub. It one needs to be tested, potentially fixed and could become our default way to uploading docker releases. This way we would store the docker hub API key in Github Action Secrets (it is there already) and individual devs do not need to bother.

--> ticket created T351720

  • Leave tarball upload as a manual process Uploading tarballs currently relies on moving SSH keys around. This is incompatible with Yubikeys. This should probably also not be moved to Github Actions as we probably are not allowed to store production keys on Github. My suggestion is to leave the process as it is for now until we spoke about the future of tarballs. If we really keep them, we will figure out a way to automatically publish them as well.

--> nothing to do here

  • Docs Update Release Checklist Phabricator Ticket Template and this document

--> ticket created T351723

  • Reproducible Builds Focus on T340226 soon.
  • Update upstream components [Spike] Find out how to update the versions of locked upstream components once our builds are reproducible.

--> ticket created T340226 to be tackled after the ones above are done

Update upstream components [Spike] Find out how to update the versions of locked upstream components once our builds are reproducible.

New ticket is T352752

roti_WMDE moved this task from In Review to Done on the Wikibase Suite Team (Sprint-∞) board.

Closed as we have follow up tasks for everything that remains.