We're currently building the same image multiple times during the patch submission/review workflow and pipeline job, and doing so on labs instances which are not secure enough. Additionally the current approach is problematic during image registration as the built image is not present on a machine we trust with registry credentials. We should refactor the pipeline to:
- isolate the image builds (both test and production variants) to secure locations (e.g. contint1001 for now)
- scale out image builders to multiple Jenkins nodes on dedicated CI hardware, each node having its own docker daemon
- use a secure intermediate docker registry that the image builders can push to and that labs CI instances can pull from and perform the subsequent pipeline steps
NOTE that the reason we're stuck on labs instances for part of the pipeline at the moment is that we're relying on minikube for the "verify" stage for isolated deployments. Once we're unsaddle with this limitation, we can ostensibly move all pipeline processes to dedicated hardware.