Page MenuHomePhabricator

Refactor pipeline build step to be more isolated/secure/scalable
Closed, InvalidPublic

Description

We're currently building the same image multiple times during the patch submission/review workflow and pipeline job, and doing so on labs instances which are not secure enough. Additionally the current approach is problematic during image registration as the built image is not present on a machine we trust with registry credentials. We should refactor the pipeline to:

  • isolate the image builds (both test and production variants) to secure locations (e.g. contint1001 for now)
  • scale out image builders to multiple Jenkins nodes on dedicated CI hardware, each node having its own docker daemon
  • use a secure intermediate docker registry that the image builders can push to and that labs CI instances can pull from and perform the subsequent pipeline steps

NOTE that the reason we're stuck on labs instances for part of the pipeline at the moment is that we're relying on minikube for the "verify" stage for isolated deployments. Once we're unsaddle with this limitation, we can ostensibly move all pipeline processes to dedicated hardware.

Event Timeline

thcipriani triaged this task as Medium priority.Jun 7 2018, 5:07 PM

We talked about this at the 2018-06-07 Deployment Pipeline meeting (adding SRE per that discussion). Specifically we talked about the need to build images that will be promoted to production through the pipeline by running the initial build on contint1001 (in a production context), tagging the resulting image, and running that through the verify step of the pipeline on CI-context machines. At the end of this process we could then tag and push from contint1001 (back in a production context).

Concerns with this plan:

  • We need to be able to clean up the registry and remove images that fail the verify step (can't do that with current registry)
  • There may be concerns over resource-contention
  • We may be able to only build and promote given some trigger for images that are specifically for release to production

The current path forward is to follow the plan roughly outlined above to avoid prematurely optimizing the pipeline for the first few services to move to using it.

Let's use this task to talk about the ideal way forward.

Vvjjkkii renamed this task from Refactor pipeline build step to be more isolated/secure/scalable to 9ocaaaaaaa.Jul 1 2018, 1:09 AM
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
thcipriani renamed this task from 9ocaaaaaaa to Refactor pipeline build step to be more isolated/secure/scalable.Jul 1 2018, 7:28 PM
thcipriani updated the task description. (Show Details)
CommunityTechBot lowered the priority of this task from High to Medium.Jul 5 2018, 6:44 PM

The model of execution that this task refers to is not longer used/valid.