Page MenuHomePhabricator

Allow use of copied artifacts in subsequent build step
Closed, ResolvedPublic

Description

Blubber outputs artifact instructions in the Install phase. It outputs builder instructions in the PreInstall phase. This combination means that you cannot describe a multi-stage Dockerfile using Blubber where one variant builds a collection of artifacts that are needed by a subsequent variant's build instruction.

My current use case for this is splitting up multiple builder commands to reduce layer invalidation. Variant A runs a builder to download and install a utility program that will not change often. This utility is needed in Variant B's builder to install other components.

Event Timeline

bd808 created this task.Sep 22 2020, 10:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 22 2020, 10:28 PM

One possible fix for this would be adding support for builder.requirements to support cross variant copying:

variants:
  A:
    lives:
      in: $SOME_DIR
    builder:
      command: [...]
  B:
    lives:
      in: $OTHER_DIR
    builder:
      requirements:
        - file1
        - file2
        - from: A
          source: $SOME_DIR
          destination: $SOME_DIR
      command: [$SOME_DIR/bin/tool, run]

I worked around this issue for Toolhub by adding new functionality to Blubber itself, but I think the basic issue is still valid. Extending the Blubber golang code is a lot more work than being able to define a graph of variants where one variant downloads or compiles a tool and a subsequent variant copies the output and then does some additional operations on that output using a builder step.

One possible fix for this would be adding support for builder.requirements to support cross variant copying:

variants:
  A:
    lives:
      in: $SOME_DIR
    builder:
      command: [...]
  B:
    lives:
      in: $OTHER_DIR
    builder:
      requirements:
        - file1
        - file2
        - from: A
          source: $SOME_DIR
          destination: $SOME_DIR
      command: [$SOME_DIR/bin/tool, run]

I like it! A couple of thoughts about implementation to make it similar to how copies is currently implemented:

  1. Normalize the data structure. In other words, make all members of requirements effectively become { from: $variant, source: $src, destination: $foo } once parsed.
  2. With the normalized structure, - file1 becomes shorthand config for - { from: local, source: file1, destination: ./ }
  3. You might actually be able to use the ArtifactsConfig for the above. In other words, do BuilderConfig's Requirements []ArtifactsConfig. I think that should work.
dduvall added a comment.EditedDec 8 2020, 12:15 AM

You might actually be able to use the ArtifactsConfig for the above. In other words, do BuilderConfig's Requirements []ArtifactsConfig. I think that should work.

Strike that. It should be CopiesConfig which is already a derivation of []ArtifactsConfig with a custom unmarshaler to handle the shorthand config.

bd808 added a comment.Dec 8 2020, 4:57 AM

You might actually be able to use the ArtifactsConfig for the above. In other words, do BuilderConfig's Requirements []ArtifactsConfig. I think that should work.

Strike that. It should be CopiesConfig which is already a derivation of []ArtifactsConfig with a custom unmarshaler to handle the shorthand config.

CopiesConfig is almost the right magic. It has a flaw however in its handling of the shorthand notation. In the case of copies, a shorthand entry like [foo] expands to {From: foo}. For use with a builder, we need [foo] to expand to {From:local, Source:foo, Destination:foo}.

CopiesConfig is almost the right magic. It has a flaw however in its handling of the shorthand notation. In the case of copies, a shorthand entry like [foo] expands to {From: foo}. For use with a builder, we need [foo] to expand to {From:local, Source:foo, Destination:foo}.

Ah, right. An additional subtype of []ArtifactsConfig would make more sense then.

Gilles added a subscriber: Gilles.Dec 8 2020, 6:59 PM

Change 647120 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[blubber@master] builder: support cross variant copying for builder.requirements

https://gerrit.wikimedia.org/r/647120

bd808 triaged this task as High priority.Dec 9 2020, 6:04 PM
bd808 moved this task from Backlog to In Progress on the Toolhub board.Dec 10 2020, 10:28 PM

Change 647120 merged by jenkins-bot:
[blubber@master] builder: support cross variant copying for builder.requirements

https://gerrit.wikimedia.org/r/647120

bd808 closed this task as Resolved.Dec 11 2020, 12:50 AM

Mentioned in SAL (#wikimedia-releng) [2020-12-11T21:32:12Z] <marxarelli> deploying blubberoid:2020-12-11-212149-production (refs https://gerrit.wikimedia.org/r/c/blubber/+/647120 and T263597)

Change 648362 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[blubber@master] requirements: Fix regression in short form handling

https://gerrit.wikimedia.org/r/648362

Change 648362 merged by jenkins-bot:
[blubber@master] requirements: Fix regression in short form handling

https://gerrit.wikimedia.org/r/648362