Implement GH actions based release automation
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	roti_WMDE
	Nov 21 2023, 2:34 PM

Description

Current Situation:

The documentation says, in order to push a release to Dockerhub, we need to download artifacts from Github actions and push them to docker hub from a dev machine. There is a docker hub upload action in our .github/workflows folder and Dockerhub secrets are configured in Github actions secrets though.

Goal:

Review and potentially fix the Github Dockerhub upload action.

Acceptance Criteria:

We can upload builds from Github CI to Dockerhub by simply using the Github UI

[optional] Notes:
[optional] Open Questions:

Does the tagging of versions on docker images work properly?
Do we want the tagging of versions to be moved next to the docker build invocation?

Related Objects

Mentioned In: T272480: Create Verification Step for GitHub workflows
T352963: ghcr loads artifacts twice
T345690: Github artifacts expire
T353732: Speed Up Builds CI runs
T351723: Release Docs Update
T343433: Automate the wbSuite Release Process
Mentioned Here: T272480: Create Verification Step for GitHub workflows
T352963: ghcr loads artifacts twice
T353732: Speed Up Builds CI runs
T345690: Github artifacts expire
T357623: remove tarball build step

Event Timeline

roti_WMDE created this task.Nov 21 2023, 2:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 21 2023, 2:34 PM

roti_WMDE updated the task description. (Show Details)Nov 21 2023, 3:24 PM

roti_WMDE updated the task description. (Show Details)

roti_WMDE added a project: Speedier Release.

darthmon_wmde mentioned this in T343433: Automate the wbSuite Release Process.Nov 21 2023, 3:39 PM

We definitely need an dockerhub user account that can be used by that action. The old account "wikibasetravis" seems to be closed.

roti_WMDE mentioned this in T351723: Release Docs Update.Dec 5 2023, 1:54 PM

roti_WMDE renamed this task from Review Docker Hub Upload Action to Review and fix Docker Hub Upload Action.Dec 5 2023, 2:09 PM

roti_WMDE set the point value for this task to 5.

roti_WMDE changed the task status from Open to In Progress.Feb 5 2024, 11:21 AM

roti_WMDE claimed this task.

roti_WMDE moved this task from To do (prio from top to bottom) to Doing on the Wikibase Suite Team (Sprint-∞) board.

partial PR https://github.com/wmde/wikibase-release-pipeline/pull/578

WIP workflow rework thoughts

This new approach also allows us to use build results always directly within the same Github action run. So we do not need to handle github action run_id numbers manually, no need to record them manually in our CHANGES.md. Also, all script for gh artifact downloading can go away.

We also do not rely on github artifacts for release build storage, fixing T345690.

Since the removal of the tarball build step T357623 we could in the future also only use GHCR container registry for (intermediate) build result storage, probably accelerating the whole pipeline a lot.

roti_WMDE renamed this task from Review and fix Docker Hub Upload Action to Implement GH actions based release automation.Feb 20 2024, 2:21 PM

PR is up for review https://github.com/wmde/wikibase-release-pipeline/pull/583

This PR adds automatic release git tagging and dockerhub upload to our CI pipeline.

This is implemented by using three CI workflows:

🧪 Build and Test main

Runs on push to main branch
Runs build
Runs test
Pushes dev images to GHCR

🏗️ Build and Test PR

Runs on every PR
On release prep PRs too
Runs build
Runs tests
Pushes dev images to GHCR

📦 Build Test Tag and Publish Release

Runs on push to mw- branch
Runs build
Runs tests
Creates git tag for release
Pushes images to dockerhub

Notable changes in this PR:

refactor CI code to improve code reuse
do not use Github artifacts for docker image storage anymore
- use the Github container registry (GHCR) instead
- this brings build times down as uploads to GHCR are way faster
- tag GHCR containers with "dev" prefix, they are not releases
- tag GHCR container with version string, branch name and run id
export images to tarball functionality is completely removed
- this again brings the build times down as no tarballs are written to disk anymore
image version number generation is centralized to reuse it in build.sh and CI
deprecated code and docs regarding artifacts, pipeline and publishing are removed
image security scanning is temporarily disabled as we do not use it

Related issues

This PR would close T345690, T352963, T353732, T272480 too

roti_WMDE moved this task from Doing to In Review on the Wikibase Suite Team (Sprint-∞) board.Feb 21 2024, 10:59 AM

roti_WMDE closed this task as Resolved.Feb 26 2024, 6:07 PM

roti_WMDE moved this task from In Review to Done on the Wikibase Suite Team (Sprint-∞) board.

roti_WMDE mentioned this in T353732: Speed Up Builds CI runs.Feb 28 2024, 3:22 PM

roti_WMDE mentioned this in T345690: Github artifacts expire.

roti_WMDE mentioned this in T352963: ghcr loads artifacts twice.Feb 28 2024, 3:29 PM

roti_WMDE mentioned this in T272480: Create Verification Step for GitHub workflows.

RickiJay-WMDE moved this task from Sprint-∞ to Archive (closed) on the Wikibase Suite Team board.Mar 5 2024, 1:02 PM

RickiJay-WMDE edited projects, added Wikibase Suite Team; removed Wikibase Suite Team (Sprint-∞).

	F42018754: Release Branches.png
	Feb 21 2024, 10:24 AM

	F41913878: image.png
	Feb 15 2024, 4:08 PM

Implement GH actions based release automationClosed, ResolvedPublic5 Estimated Story PointsActions