Page MenuHomePhabricator

Blubber "copies" and "builder command" steps should run in the opposite order
Closed, DeclinedPublic

Description

Defining such a variant:

dev:
    copies: [build]
    builder:
      command:
        ["ls", "-al"]

Results in the builder command being run before artifacts from the other build are copied over:

Step 23/28 : RUN ls "-al"
 ---> Running in 9147b648bf85
total 8
drwxr-xr-x 2 somebody somebody 4096 Dec  2 10:20 .
drwxr-xr-x 1 root     root     4096 Dec  2 10:20 ..
Removing intermediate container 9147b648bf85
 ---> a11db376693a
Step 24/28 : COPY --chown=65533:65533 --from=build ["/srv/app", "/srv/app"]

As such, there is no way to run a builder command that depends on the local files.

The context is that I'm trying to reproduce this Dockerfile in Blubber: https://github.com/gi11es/thumbor-docker/blob/main/Dockerfile#L34

The highlighted line is one of the problematic steps to reproduce. It generates a random key file. This can't be run as the builder command of a variant, because it's multiple programs piped. I thought that as a workaround I could write a local Makefile instead, and call that from the builder command. Alas, the local Makefile won't get copied over to the container until after the builder command has run.

It would seem more logical to me that when defining that a variant copies files from another or from "local", that this steps happens before builder commands, as it would allow to access custom local files to build things.

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedakosiaris
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedBawolff
ResolvedAnomie
ResolvedBawolff
ResolvedBawolff
ResolvedLegoktm
ResolvedLucas_Werkmeister_WMDE
ResolvedBawolff
Resolvedsbassett
Resolvedsbassett
ResolvedJdforrester-WMF
Resolvedsbassett
Resolvedsbassett
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
Resolvedhashar
Resolvedhashar
ResolvedJdforrester-WMF
Resolvedhashar
DeclinedMoritzMuehlenhoff
Invalidthcipriani
Resolvedmmodell
Resolvedhashar
ResolvedJoe
ResolvedJMeybohm
ResolvedJMeybohm
DuplicateDzahn
DeclinedDzahn
ResolvedJdforrester-WMF
OpenNone
OpenNone
OpenGilles
DeclinedNone

Event Timeline

As such, there is no way to run a builder command that depends on the local files.

The context is that I'm trying to reproduce this Dockerfile in Blubber: https://github.com/gi11es/thumbor-docker/blob/main/Dockerfile#L34

The highlighted line is one of the problematic steps to reproduce. It generates a random key file. This can't be run as the builder command of a variant, because it's multiple programs piped. I thought that as a workaround I could write a local Makefile instead, and call that from the builder command. Alas, the local Makefile won't get copied over to the container until after the builder command has run.

For local files, that's what builder.requirements is for. In your case, something like this may work.

variants:
  dev:
    copies: [build]
    builder:
      command: [make, thumborkey]
      requirements: [Makefile]

There does seem to be a limitation around running builders against artifacts from other variants, however, and the ordering of things is non-obvious.

It would seem more logical to me that when defining that a variant copies files from another or from "local", that this steps happens before builder commands, as it would allow to access custom local files to build things.

The purpose of having builders run before copying of local (or variant) files is to improve the cacheability of image layers. However, this does seem to be a point of confusion for many users. It seems we'll need to address this somehow, perhaps refactoring the builder configuration to make ordering of both file copying and command running more explicit.

I forgot to suggest this as another option in solving your specific use case since you mentioned needing to pipe output.

variants:
  dev:
    copies: [build]
    builder:
      command: [sh, -c, "dd if=/dev/urandom of=- bs=1024 count=1 2>/dev/null | md5sum -b - | cut -d' ' -f1 > thumbor.key"]

Of course, if you have more commands to run, a Makefile or similar might be a better option.

I think this ticket is functionally a duplicate of T263597: Allow use of copied artifacts in subsequent build step.

I just smashed my face into this problem again with the Toolhub project. I need to run a webpack builder step in a container with a nodejs runtime and then copy artifacts generated by webpack into a container with a python runtime before running a Django admin command (collectstatic). There is no apparent method for doing this using Blubber today.

I'm wondering if my idea from T263597#6485729 would be accepted as a patch if I spent the time to write it. I think allowing builder.requirements to copy from other variants would help preserve cache efficiency while enabling interdependent build steps.

Declining since this was specifically scoped to changing the behavior of copies which we can't really do without breaking core functionality, and it seems T263597: Allow use of copied artifacts in subsequent build step will meet the needs of the use case.