Page MenuHomePhabricator

Support copying files from run containers into the workspace for use in other stages
Closed, ResolvedPublic

Description

During pipeline office hours, we brainstormed about a new feature that would be helpful for users wanting to process files that were created as a result of running the container in a previous stage. We could conceivably implement a feature that was already stubbed out for future use, namely the publish.files configuration.

Example:

stages:
  - name: test
    build: test
    run:
      image: true
    publish:
      files:
        - from: '${.container}'
          source: coverage.xml
          destination: coverage.xml

In the above example, the file coverage.xml is copied from the stopped container of the current stage into the workspace. The ${.container} context variable would reference the name of the stopped container, and source and destination would reference the file source within the container filesystem and destination relative to the working directory. Implementation would likely use docker cp container:path path.

Subsequent stages may then copy that file during the build step and process it from the entrypoint or a builder command.

Event Timeline

Change 674404 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/pipelinelib@master] Support publish.files for copying files from run containers

https://gerrit.wikimedia.org/r/674404

Change 674404 merged by jenkins-bot:
[integration/pipelinelib@master] Support copy for copying files from containers to the local context

https://gerrit.wikimedia.org/r/674404

Change 787692 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[research/mwaddlink@main] CI: Copy coverage reports from test to codehealth stage

https://gerrit.wikimedia.org/r/787692

Change 787692 abandoned by Kosta Harlan:

[research/mwaddlink@main] CI: Copy coverage reports from test to codehealth stage

Reason:

squashed

https://gerrit.wikimedia.org/r/787692

@dduvall @thcipriani I was wondering if I could get some help in implementing this in this patch for research/mwaddlink. I've tried many invocations, but I don't seem to be able to get the files to copy.

@dduvall @thcipriani I was wondering if I could get some help in implementing this in this patch for research/mwaddlink. I've tried many invocations, but I don't seem to be able to get the files to copy.

Here's the relevant (open) task: T307209: mwaddlink CI fails with "No module named pip"