I've outlined what it would take to deprecate the CLI and blubberoid, but it's important to note this isn't a done deal and there isn't a timeline yet. I want to leave room for discussion.

@bd808, bringing you into the discussion since I know you are or have relied on the CLI and/or blubberoid in some of your developer tooling. Do you have opinions on deprecation in general and/or thoughts on a timeline? IIRC, you're using docker compose which should be able to use the BuildKit frontend and reference a blubber.yaml directly now so long as there's a # syntax = line present, but I haven't tested this at all.

In T318289#8253975, @dduvall wrote:

@bd808, bringing you into the discussion since I know you are or have relied on the CLI and/or blubberoid in some of your developer tooling. Do you have opinions on deprecation in general and/or thoughts on a timeline? IIRC, you're using docker compose which should be able to use the BuildKit frontend and reference a blubber.yaml directly now so long as there's a # syntax = line present, but I haven't tested this at all.

Using a published front-end container is actually easier than dealing with Blubberoid I think, so no complaints from me about deprecation. I think I can migrate Striker, Wikimedia-Developer-Portal, and Toolhub relatively quickly. T318631: Migrate Striker local dev away from blubberoid generated Dockerfiles is actually already done and the other two should be a similar amount of work.

In T318289#8262839, @bd808 wrote:

I think I can migrate Striker, Wikimedia-Developer-Portal, and Toolhub relatively quickly.

I jinxed myself with that statement. See T318866: "qemu: uncaught target signal 11" building local dev container on M1 Mac with Docker Desktop for an annoyingly special problem that has shown up while converting from a statically cached Dockerfile generated by blubberoid to use of the blubber.yaml with a blubber-buildkit frontend comment for the Wikimedia-Developer-Portal dev environment. In theory both methods should be producing identical Dockerfile instructions, but the qemu crash seems to indicate that something is varying in an unexpected (and bug triggering) way.

• Refactor Blubber internally to construct its build graph using BuildKit LLB and remove Dockerfile compiler

This being the final step in the checklist may be causing a chicken-and-egg deadlock problem in progressing. T318866: "qemu: uncaught target signal 11" building local dev container on M1 Mac with Docker Desktop seems to be a bug related to the Dockerfile to LLB transcoding currently used to enable use as a buildkit frontend:

In T318866#8360318, @dduvall wrote:

In T318866#8360111, @bd808 wrote:

Reading T321316: Self-build and publish buildkit helper images clued me into the copy action where the crash is happening being very likely done via the docker/dockerfile-copy image which is used as a helper by buildkit.

Just for clarity: It's used as a helper by buildkit's dockerfile frontend, specifically the dockerfile2llb package, to implement Dockerfile ADD and COPY operations.

Once we fully deprecate transcoding to Dockerfile syntax in Blubber (blubberoid and CLI transcoding) and refactor the build instructions to use native LLB operations, Blubber will no longer be dependent on dockerfile2llb and the dockerfile/copy helper image it uses.

The easiest way to to break this dependency loop from the Blubber perspective would probably be to declare that ARM runtime support is not in scope. The developer-portal project's use of Blubber to generate a Dockerfile for local development would then be an unsupported dependency that the project needs to figure out independent of Blubber.

Another way to attempt to break the cycle would be to implement a native LLB generation mode in Blubber in parallel with the Dockerfile generation mode. This would allow testing to see if T318866 is resolved by that change.

In T318289#8404147, @bd808 wrote:

Another way to attempt to break the cycle would be to implement a native LLB generation mode in Blubber in parallel with the Dockerfile generation mode. This would allow testing to see if T318866 is resolved by that change.

I went this way with it. The native-LLB refactor has merged to an experimental branch and there's a newly published image that is free of Dockerfile and its inherent limitations such as the dockerfile-copy helper image. Try it out!

docker-registry.wikimedia.org/repos/releng/blubber/buildkit:experimental-native-llb

In T356684#9522428, @dduvall wrote:

In T356684#9522131, @taavi wrote:

In T356684#9522127, @dduvall wrote:

Oh, you're using Blubberoid! It's deprecated and I'm planning to decommission it soon. Can you point me to your downstream project? Can we get that using the Blubber buildkit frontend perhaps (# syntax = line in the blubber.yaml?

Oops. I must've missed the announcement of that. Using the buildkit syntax seems to work, indeed.

No, you didn't. I never announced it. :) I had the very incorrect assumption that no one was using Blubberoid outside of CI.

Glad it works now!

Per https://codesearch.wmcloud.org/search/?q=blubberoid Shellbox and MediaWiki-Vagrant are still using blubberoid.wikimedia.org in some fashion. Separately I know that the developer-portal dev environment also uses blubberoid because of T318632: Migrate developer-portal local dev away from blubberoid generated Dockerfiles. I think I can move developer-portal and Shellbox to the new method now per my testing in T318866#9438974.

Change 998560 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[mediawiki/libs/Shellbox@master] dev: Replace blubberoid with blubber buildkit

https://gerrit.wikimedia.org/r/998560

Thanks for tackling Shellbox, @bd808. It looks like MW-Vagrant's Blubberoid wrapper is in turn used by https://gerrit.wikimedia.org/g/mediawiki/vagrant/+/4b15d7aeb71e28b13a653f137dc18bec0376db8d/puppet/modules/role/templates/addimage/systemd/image-suggestion-api.epp

Change 1002589 had a related patch set uploaded (by Dduvall; author: Dduvall):

[mediawiki/vagrant@master] Remove use of Blubberoid

https://gerrit.wikimedia.org/r/1002589

Change 1002589 merged by jenkins-bot:

[mediawiki/vagrant@master] Remove use of Blubberoid

https://gerrit.wikimedia.org/r/1002589

Change 998560 merged by jenkins-bot:

[mediawiki/libs/Shellbox@master] dev: Replace blubberoid with blubber buildkit

https://gerrit.wikimedia.org/r/998560

Mentioned in SAL (#wikimedia-operations) [2024-05-23T16:37:33Z] <dduvall> destroying all blubberoid deployments as part of its decommissioning (T318289)

Regarding

• (main branch) Remove cmd/blubber and cmd/blubberoid interfaces from Blubber

I was recently doing development on Blubber and found the blubber command to be a very convenient debugging tool. I hope that something remains that allows for determining "what instructions does this blubber.yaml file turn into?".

In T318289#9826786, @dancy wrote:

I was recently doing development on Blubber and found the blubber command to be a very convenient debugging tool. I hope that something remains that allows for determining "what instructions does this blubber.yaml file turn into?".

+1 for some method of dumping the "compiled" version to understand what is actually going to happen when the container is built. It does seem that inspecting the layers of generated image produces a fairly readable output though, so maybe we could get away with just some documentation of how to extract that metadata?

$ docker history toolhub:dev-python --format json --no-trunc | jq .CreatedBy --raw-output | tac
/bin/sh -c #(nop) ADD file:8f70dbd0b0122c3ef83035a8c0cc1fd31a32e7a00942edc583768fa4e829d774 in /
/bin/sh -c #(nop)  ENV LC_ALL=C.UTF-8
/bin/sh -c #(nop)  CMD ["/bin/bash"]
|0 /bin/sh -c echo 'Acquire::http::Proxy "http://webproxy.codfw.wmnet:8080";' > /etc/apt/apt.conf.d/80_proxy     && apt-get update      && DEBIAN_FRONTEND=noninteractive     apt-get install  --yes python3 python3-pip python3-setuptools --no-install-recommends     && rm -f /etc/apt/apt.conf.d/80_proxy     && apt-get clean && rm -rf /var/lib/apt/lists/*
USER 0
ENV HOME=/root
ENV DEBIAN_FRONTEND=noninteractive
RUN /bin/sh -c apt-get update && apt-get install -y "build-essential" "pkg-config" "default-libmysqlclient-dev" "gettext" "git" "python3-dev" "python3-venv" && rm -rf /var/lib/apt/lists/* # buildkit
ARG LIVES_AS=somebody
ARG LIVES_UID=65533
ARG LIVES_GID=65533
RUN |3 LIVES_AS=somebody LIVES_UID=502 LIVES_GID=20 /bin/sh -c (getent group "$LIVES_GID" || groupadd -o -g "$LIVES_GID" -r "$LIVES_AS") && (getent passwd "$LIVES_UID" || useradd -l -o -m -d "/home/$LIVES_AS" -r -g "$LIVES_GID" -u "$LIVES_UID" "$LIVES_AS") && mkdir -p "/srv/app" && chown "$LIVES_UID":"$LIVES_GID" "/srv/app" && mkdir -p "/opt/lib" && chown "$LIVES_UID":"$LIVES_GID" "/opt/lib" # buildkit
ARG RUNS_AS=runuser
ARG RUNS_UID=900
ARG RUNS_GID=900
RUN |6 LIVES_AS=somebody LIVES_UID=502 LIVES_GID=20 RUNS_AS=runuser RUNS_UID=900 RUNS_GID=900 /bin/sh -c (getent group "$RUNS_GID" || groupadd -o -g "$RUNS_GID" -r "$RUNS_AS") && (getent passwd "$RUNS_UID" || useradd -l -o -m -d "/home/$RUNS_AS" -r -g "$RUNS_GID" -u "$RUNS_UID" "$RUNS_AS") # buildkit
USER 502
ENV HOME=/home/somebody
WORKDIR /srv/app
ENV DJANGO_SETTINGS_MODULE=toolhub.settings PIP_DISABLE_PIP_VERSION_CHECK=on PIP_NO_CACHE_DIR=off PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
COPY pyproject.toml poetry.lock ./ # buildkit
RUN |6 LIVES_AS=somebody LIVES_UID=502 LIVES_GID=20 RUNS_AS=runuser RUNS_UID=900 RUNS_GID=900 /bin/sh -c python3 "-m" "venv" "/opt/lib/venv" # buildkit
ENV PATH=/opt/lib/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin VIRTUAL_ENV=/opt/lib/venv
RUN |6 LIVES_AS=somebody LIVES_UID=502 LIVES_GID=20 RUNS_AS=runuser RUNS_UID=900 RUNS_GID=900 /bin/sh -c python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "tox" "pip" # buildkit
ENV POETRY_VIRTUALENVS_PATH=/opt/lib/poetry
RUN |6 LIVES_AS=somebody LIVES_UID=502 LIVES_GID=20 RUNS_AS=runuser RUNS_UID=900 RUNS_GID=900 /bin/sh -c python3 "-m" "pip" "install" "-U" "poetry==1.3.1" # buildkit
RUN |6 LIVES_AS=somebody LIVES_UID=502 LIVES_GID=20 RUNS_AS=runuser RUNS_UID=900 RUNS_GID=900 /bin/sh -c mkdir -p "/opt/lib/poetry" # buildkit
RUN |6 LIVES_AS=somebody LIVES_UID=502 LIVES_GID=20 RUNS_AS=runuser RUNS_UID=900 RUNS_GID=900 /bin/sh -c poetry "install" "--no-root" # buildkit
COPY . . # buildkit
COPY /srv/dockerize /srv/dockerize # buildkit
LABEL blubber.variant=local-python blubber.version=0.23.0+052b773

In T318289#9827420, @bd808 wrote:

In T318289#9826786, @dancy wrote:

I was recently doing development on Blubber and found the blubber command to be a very convenient debugging tool. I hope that something remains that allows for determining "what instructions does this blubber.yaml file turn into?".

+1 for some method of dumping the "compiled" version to understand what is actually going to happen when the container is built. It does seem that inspecting the layers of generated image produces a fairly readable output though, so maybe we could get away with just some documentation of how to extract that metadata?

The blubber cmd remains in the native-LLB version and outputs protobuf encoded LLB which can be fed to buildctl build or buildctl debug dump-llb. The latter is JSON by default but can be a DOT digraph for viewing of the overall build graph.

In T318289#9927401, @dduvall wrote:

In T318289#9827420, @bd808 wrote:

In T318289#9826786, @dancy wrote:

I was recently doing development on Blubber and found the blubber command to be a very convenient debugging tool. I hope that something remains that allows for determining "what instructions does this blubber.yaml file turn into?".

+1 for some method of dumping the "compiled" version to understand what is actually going to happen when the container is built. It does seem that inspecting the layers of generated image produces a fairly readable output though, so maybe we could get away with just some documentation of how to extract that metadata?

The blubber cmd remains in the native-LLB version and outputs protobuf encoded LLB which can be fed to buildctl build or buildctl debug dump-llb. The latter is JSON by default but can be a DOT digraph for viewing of the overall build graph.

Please add this info to README.md (and maybe CONTRIBUTING.md).