@dcausse thanks for filing this.

Yes, I've been trying to get Blubber's multiplatform image published for a while now, but the problem there was related to the config of our registry's nginx frontend, a separate issue from the one you're seeing.

In BuildKit frontend gateways—the component responsible for ingesting the user provided image config and turning that into low level build (LLB) instructions—there is a build platform and a target platform. The former is inferred from the first available worker in buildkitd's worker pool. The latter is provided by the client. In our case, the build platform is always going to be linux/amd64 unless we roll out CI nodes that are of a different arch which we're not planning to.

The error you're seeing occurs when a binary is executed during the build process that is of a different arch than the build platform and the binfmt_misc module is not loaded and configured. I'm not certain, but I believe this must be on the build container's host system or with sufficient privileges to /proc/sys/fs, which is what your use of docker run --privileged --rm tonistiigi/binfmt --install all and the buildkit docs suggest. This would take some research and experimentation to see if we could get that working on the Digital Ocean cloud and/or WMF trusted runners.

In the meantime, the reason our Blubber multi-platform build works is because it relies solely on cross-compiling and doesn't need to _execute_ any non-amd64 binaries. It makes use of the environment variables exposed to build processes by BuildKit that specify details of both the build platform and target platform (OS, arch, arm64 revision, etc.).

See https://gitlab.wikimedia.org/repos/releng/blubber/-/blob/0734952947504560483c65edbc7f0da4634dfbdb/Makefile#L22 and https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope

I hope that helps in the case you're able to cross compile ES without emulation, but in either case I'd be down to pair on this with you and see if we can't get some solution working.

@dduvall thanks for the clarifications this does help a lot!

My build is very minimal as everything is already built (java jar files and thus already multi-arch) via another repo and we just need to unzip a deb package in particular folder.
I tried to move all the custom building steps into a buildvariant and then use the "copies" keyword for the production variant (the one I want to be multi-arch) but I still have troubles with:

• blubber lives and runs config blocks seem to generate a command like:
  • RUN (getent group "65533" || groupadd -o -g "65533" -r "somebody") && (getent passwd "65533" || useradd -l -o -m -d "/home/somebody" -r -g "65533" -u "65533" "somebody") && mkdir -p "/usr/share/elasticsearch/plugins" && chown "65533":"65533" "/usr/share/elasticsearch/plugins" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"
  • which is I think executed within the arm64 image and then are causing the error: standard_init_linux.go:219: exec user process caused: exec format error. My understanding is that if I want to use copies I must be setting lives.in, in my cases all the folders/users are existing, would there be a way to skip this detection and tell blubber to trust my image?

• I'm not quite clear on how to define the arch with a multi-stage build scenario, what I understand I need is:
  • run the build variant on amd64
  • reuse that variant and build the production variant for arm64 and amd64

I tried this with kokkuri:

test-arm64:
  extends: .kokkuri:build-image
  stage: test
  variables:
    BUILD_VARIANT: production
    BUILD_TARGET_PLATFORMS: linux/arm64

For reference https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-elasticsearch-image/-/merge_requests/2 is the MR where I tried to switch from using the include to the copies logic to minimize the commands ran when building the production image.

I'd be happy to pair on this if you have time :)

Thanks!

In T334254#8772309, @dcausse wrote:

• which is I think executed within the arm64 image and then are causing the error: standard_init_linux.go:219: exec user process caused: exec format error. My understanding is that if I want to use copies I must be setting lives.in, in my cases all the folders/users are existing, would there be a way to skip this detection and tell blubber to trust my image?

This is making a lot more sense to me now. Another aspect to why the blubber multi-platform build is probably working and this isn't is that blubber's variant is based on a scratch image (no base), and user/group creation is skipped in that case. So yeah, if we had an option to skip the user/group creation, that would be another viable workaround.

The better long-term solution is still likely going to be getting binfmt_misc working, but we can tackle this from a couple of different angles.

I'd be happy to pair on this if you have time :)

Thanks!

No problem! I don't have much time this week, but I'll throw something on the calendar for next week. Feel free to suggest other days/times. It doesn't seem like we have much overlap in working hours, but I'm sure we'll figure something out. :)