Page MenuHomePhabricator

Define pipeline failure developer feedback
Open, MediumPublic

Description

As a patch moves through the deployment pipeline, the test image will be built and run - this will either succeed or fail. The production image will be built and pushed - this will either succeed or fail. The failures need to be surfaced to developers in an actionable way if they are the result of user-error.

Event Timeline

In talking with @dduvall the ideal would be, on failure, providing a url that could be passed to docker pull that lets the developer pull down an image that didn't pass muster.

thcipriani triaged this task as Medium priority.Jul 16 2018, 7:31 PM

Change 490640 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] Gerrit: comment styles for the pipeline

https://gerrit.wikimedia.org/r/490640

Change 490640 merged by Alexandros Kosiaris:
[operations/puppet@production] Gerrit: comment styles for the pipeline

https://gerrit.wikimedia.org/r/490640

In talking with @dduvall the ideal would be, on failure, providing a url that could be passed to docker pull that lets the developer pull down an image that didn't pass muster.

For a url to be provided to the end developer, a container image registry is required.

Without some aggressive GC (one which we currently don't have) allowing us to delete images on a rather short time window after they are created, this is bound to cause operational issues. Until we have such functionality available, it probably is more appropriate to provide them with the base docker image and the resulting Dockerfile to allow them to build the image and run the tests locally.

In addition to failures, it'd be nice to know when a tested and built image is ready and available for deployment.

We currently have 2 paths to an image being built and pushed:

  1. Gerrit patch merging, triggers a "post-merge" job to build and push an image. In that case, a reasonable point of communication would be the Gerrit patch.
  2. Repo author pushes tag, triggers a "ref-update" job that builds and pushes an image that is tagged whatever that tag is. In that instance, a reasonable point of communication would probably have to be email.

Via IRC:

15:28:30 thcipriani | tags triggering an email seems like a reasonable next thing to implement. Would it just send to the tag author?
15:29:35   ottomata | could be fine, but maybe it'd be more flexible to configure the alert emails also in .pipeline/