Page MenuHomePhabricator

Kokkuri should allow dockerfile.v0 frontend
Closed, ResolvedPublic

Description

In a recent Slack conversation with the releng team, we discussed how to get rid of a custom gitlab runner that was put in place on T321736. This custom runner allows us to run 'docker besides docker', with the final goal of building and running a Dockerfile that generates an production artifact.

Instead of running 'docker besides docker', we live coded a proof of concept on using buildkit to build and extract the artifact without the need of 'docker besides docker'.

Here are the draft MR with the proof of concept:
https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/36
https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/17

In this task we want to:

  1. Generate a custom gitlab runner image that exposes buildkit with a more user friendly interface for building and extracting an artifact.
  2. Test that such mechanism can replace T321736.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Use kokkuri v1.6.0 imagerepos/releng/kokkuri!71dancyreview/dancy/use-v1.6.0-imagemain
Allow docker/dockerfile frontendrepos/releng/kokkuri!70dancyreview/dancy/allow-dockerfilemain
Customize query in GitLab

Event Timeline

thcipriani renamed this task from Gitlab runner image for building and extracting an artifact via a Dockerfile. to Kokkuri should allow dockerfile.v0 frontend.Feb 1 2023, 5:44 PM

Hello there. Checking if this task is still on releng's radar, and whether there is an ETA?

I recently had to do some maintenance on my custom gitlab runner and so was reminded of this.

Thanks!

@xcollazo Thanks for the ping. I'll work on getting something going this week.

@xcollazo I've updated kokkuri to support Dockerfiles. Sample use:

include:
  - project: 'repos/releng/kokkuri'
    file: 'includes/images.yaml'

stages:
  - one

test:
  stage: one
  extends: .kokkuri:build-image
  variables:
    BUILD_CONFIG: Dockerfile
    BUILD_VARIANT: ""

Point BUILD_CONFIG at your dockerfile and set BUILD_VARIANT to the name of the build stage or to a blank string to indicate the default stage.

Lemme know how it goes.

Btw, the dockerfile frontend does not work right with .kokkuri:build-and-run-image because the run-image logic is only implemented by the blubber/buildkit frontend.

Awesome! Thanks @dancy, will test this in the near future!

For completeness, we did start utilizing this mechanism via @Antoine_Quhen's work on T318346 for our airflow-dags project. We also want to migrate conda-analytics to this mechanism in the future.

So, thanks @dancy !