Page MenuHomePhabricator

Add new syntax directive to blubber.yaml files to enable users to directly use docker build with blubber.yaml.
Closed, ResolvedPublic

Description

A few days ago, I was using blubber as a microservice on the ML sandbox. Today, I tried using the same workflow but kept getting an uninformative error message. When I referenced the docs, they had changed.

We may no longer have to use blubber as a microservice, we can use Docker build tools with syntax directive described here: https://wikitech.wikimedia.org/wiki/Blubber/Download#Blubber_as_a_BuildKit_frontend

This means that the blubber.yaml files that we use to build all our Docker images might have to change and include this new syntax directive.

Event Timeline

kevinbazira renamed this task from Add new syntax directive to blubber.yaml files to enable users to directly use docker build with blubber.yam. to Add new syntax directive to blubber.yaml files to enable users to directly use docker build with blubber.yaml..Oct 31 2022, 7:52 AM

I experimented with the syntax directive and successfully built an articlequality image using docker build with a blubber.yaml file.

Here is the workflow I used:

  1. Add directive to blubber file:
# syntax=docker-registry.wikimedia.org/repos/releng/blubber:v0.12.1
  1. Build image:
$ cd machinelearning-liftwing-inference-services
$ docker build --tag kevinbazira/articlequality-fawiki --target production --file .pipeline/articlequality/blubber.yaml .

Things to note:

  1. this directive was added to Docker in version 20.10.0.
  2. buildkit should be enabled in your environment if it is not, you could use:
$ DOCKER_BUILDKIT=1 docker build --tag kevinbazira/articlequality-fawiki --target production --file .pipeline/articlequality/blubber.yaml .

Change 856880 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] ci: add new syntax directive for blubber files

https://gerrit.wikimedia.org/r/856880

For M1 (apple silicon) processors docker build will not work with this directive
However by following the bug reported in -> https://phabricator.wikimedia.org/T318866 and the MR in -> https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/15
The following directive works - using a newer version of the buildkit image which supports multiple platforms.
# syntax=docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v0.15.0

and we build the image by specifying the target platform
docker buildx build --target production -f .pipeline/articlequality/blubber.yaml --platform=linux/amd64 . -t TAG_OF_YOUR_CHOICE

as well as
docker build --target production -f .pipeline/articlequality/blubber.yaml -t TAG_OF_YOUR_CHOICE .

Change 856880 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] ci: add new syntax directive for blubber files

https://gerrit.wikimedia.org/r/856880