Change Details

When attempting to publish a multi-platform image to the WMF container registry, `buildctl`/BuildKit fails with: `error: failed to solve: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried` See https://gitlab.wikimedia.org/repos/releng/blubber/-/jobs/28242 ``` ... #27 exporting to image #27 exporting layers #27 exporting layers 0.9s done #27 exporting manifest sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f done #27 exporting config sha256:268701062a811129e68e6d0f9a53bdd64fea1e8e083a955f4cfc98aacd591322 done #27 exporting manifest sha256:76c748e512209fca21610a44789f4911e8ac6526a93210678a44398c9fb28041 done #27 exporting config sha256:81c11b26f86c1d88d784580e9d7567714347712696a637d4066ab9e63bdb9ca0 done #27 exporting manifest list sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 done #27 pushing layers #27 pushing layers 2.2s done #27 pushing manifest for docker-registry.discovery.wmnet/repos/releng/blubber/buildkit:v0.14.0@sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 #27 pushing manifest for docker-registry.discovery.wmnet/repos/releng/blubber/buildkit:v0.14.0@sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 0.3s done #27 ERROR: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried ------ > exporting to image: ------ error: failed to solve: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried ``` This build is attempting to push a multi-platform image under a single tag, which makes use of a [[ https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list | manifest list ]] in the v2 spec that references multiple other manifests which in turn reference the platform specific layers. In order to start supporting other platforms (such as `linux/arm64` for our M1 using staff) we'll need the WMF registry to support the manifest list format. See https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list See https://github.com/opencontainers/image-spec/blob/main/image-index.md

(edit: was "WMF container registry does not accept a manifest list (aka OCI manifest index, or "fat" manifest" but retitled as this seems to be a condition on the client side) ---- When attempting to publish a multi-platform image to the WMF container registry, `buildctl`/BuildKit fails with: `error: failed to solve: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried` See https://gitlab.wikimedia.org/repos/releng/blubber/-/jobs/28242 ``` ... #27 exporting to image #27 exporting layers #27 exporting layers 0.9s done #27 exporting manifest sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f done #27 exporting config sha256:268701062a811129e68e6d0f9a53bdd64fea1e8e083a955f4cfc98aacd591322 done #27 exporting manifest sha256:76c748e512209fca21610a44789f4911e8ac6526a93210678a44398c9fb28041 done #27 exporting config sha256:81c11b26f86c1d88d784580e9d7567714347712696a637d4066ab9e63bdb9ca0 done #27 exporting manifest list sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 done #27 pushing layers #27 pushing layers 2.2s done #27 pushing manifest for docker-registry.discovery.wmnet/repos/releng/blubber/buildkit:v0.14.0@sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 #27 pushing manifest for docker-registry.discovery.wmnet/repos/releng/blubber/buildkit:v0.14.0@sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 0.3s done #27 ERROR: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried ------ > exporting to image: ------ error: failed to solve: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried ``` This may be related to an [[ https://github.com/containerd/containerd/issues/5978 | upstream bug in containerd ]]. Note that we're using OCI workers for building, not containerd. However, containerd is used to push. There seems to have been a [[ https://github.com/containerd/containerd/pull/6995 | fix merged ]] as well, so I'll see about getting this into our BuildKit build.

(edit: was "WMF container registry does not accept a manifest list (aka OCI manifest index, or "fat" manifest" but retitled as this seems to be a condition on the client side) ---- When attempting to publish a multi-platform image to the WMF container registry, `buildctl`/BuildKit fails with: `error: failed to solve: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried` See https://gitlab.wikimedia.org/repos/releng/blubber/-/jobs/28242 ``` ... #27 exporting to image #27 exporting layers #27 exporting layers 0.9s done #27 exporting manifest sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f done #27 exporting config sha256:268701062a811129e68e6d0f9a53bdd64fea1e8e083a955f4cfc98aacd591322 done #27 exporting manifest sha256:76c748e512209fca21610a44789f4911e8ac6526a93210678a44398c9fb28041 done #27 exporting config sha256:81c11b26f86c1d88d784580e9d7567714347712696a637d4066ab9e63bdb9ca0 done #27 exporting manifest list sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 done #27 pushing layers #27 pushing layers 2.2s done #27 pushing manifest for docker-registry.discovery.wmnet/repos/releng/blubber/buildkit:v0.14.0@sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 #27 pushing manifest for docker-registry.discovery.wmnet/repos/releng/blubber/buildkit:v0.14.0@sha256:9828d7ddccdf65bde8b25d05794309794c2c3b87d185fe0d00ff4824ef47f503 0.3s done #27 ERROR: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried ------ > exporting to image: ------ error: failed to solve: failed commit on ref "manifest-sha256:b1f76c874d0417382df17036e0f8a732281762fb85694d02378e3a175d1c103f": cannot reuse body, request must be retried ``` This build is attempting to push a multi-platform image under a single tamay be related to an [[ https://github.com/containerd/containerd/issues/5978 | upstream bug in containerd ]]. Note that we're using OCI workers for building, which makes use of a [[ https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list | manifest list ]] in the v2 spec that references multiple other manifests which in turn reference the platform specific layers.not containerd. However, In order to start supporting other platforms (such as `linux/arm64` for our M1 using staff) we'll need the WMF registry to support the manifest list formatcontainerd is used to push. See https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list See https://github.com/opencontainers/image-spec/blob/main/image-index.mdThere seems to have been a [[ https://github.com/containerd/containerd/pull/6995 | fix merged ]] as well, so I'll see about getting this into our BuildKit build.