Page MenuHomePhabricator

Speed Up Builds CI runs
Closed, ResolvedPublic

Description

Current Situation:

  • When building locally, Docker takes advantage of a cache system to avoid needlessly re-downloading identical dependencies. Building on CI uses no such system, resulting in repeatedly re-downloading identical dependencies, wasting time during the build process.

Goal:

  • Building on CI, particularly on test branches, should use a cache system, for the sake of both expediency and efficiency.

Acceptance Criteria:

  • Docker builds on CI use a cache system

[optional] Notes:

  • pushing and pulling from the GHCR might be enough to implement such a cache

[optional] Open Questions:

  • Are there any known issues with Docker's cache implementation that would indicate we should avoid using it for the release builds?
    • we have the option to bypass the cache with ./build.sh --no-cache if we want to be on the safe side

Event Timeline

I did a small experiment on this here. Unfortunately, pulling previous builds seems to not populate the docker build cache. At least the linked change does not speed up build times: before / after.

Interesting though: The pipeline seems to spend lots of time uploading and downloading artifacts: For example: uploading the built wdqs docker container as github artifact takes 1m50 while pushing the same container to GHCR only takes 14s. Is there a way to use the container registry in to pass docker images from build to test jobs in order to optimize CI pipeline runtimes?

roti_WMDE renamed this task from Cache Docker Images in CI to Speed Up Builds to Speed Up Builds CI runs.Feb 6 2024, 1:44 PM
roti_WMDE claimed this task.

T351720 did a decent speedup to the build. no further investigation required for now