Change Details

Adopting Zuul as a primary CI system for some GitLab hosted repos will afford us valuable gating and merging functionality (see {T349872}), but we lose CI standardization and take on the burden of supporting (managing/documenting/implementing) an additional type CI configuration. If we promote the use of `.zuul.yaml` files in individual repos, we retain CI self service. However, it's an unknown whether or not the Ansible driven configuration of Zuul is sufficient for end users, and while GitLab CI has gone through community consultation, Zuul has not. If we manage the Zuul/Ansible configuration separately from end-user repos, we lose CI self service altogether. Neither of these options seem like great outcomes. There may be an alternative, however, whereby we benefit from Zuul gating and retain CI self-service in the canonical form of `.gitlab-ci.yml` end-user configuration. 1. Integrate Zuul with GitLab for gating/merging.. 2. Manage all (or most) Zuul configuration separately from individual repos in a Zuul trusted project, including Zuul tenants, pipelines, projects, and jobs. 3. Define a standard Zuul job for all GitLab repos (e.g. `gitlab-pipeline`) that: 1. Creates a GitLab pipeline for the change (MR) using the GitLab API, passing the Zuul build UUID as a pipeline variable. 2. Exposes the speculative repo state, which includes all dependent repos, to the GitLab CI jobs via something like rsyncd/ssh/git. A shared volume may also be possible, however such an implementation would have to be possible with both k8s and Docker based GitLab runners. 3. Waits for the completion of the GL pipeline and fails/succeeds based on the result. 4. The repo-configured GitLab pipeline would in turn. 1. Configure `workflow` rules such that only Zuul triggered pipelines are run (e.g. `if: $CI_PIPELINE_SOURCE == 'api' && $ZUUL_BUILD`) 2. Sync the speculative repo state from the Zuul executor (`/var/lib/zuul/builds`) (or node `/src`) using rsync or git in either a `before_script` or `hooks:pre_get_sources_script` [[ https://docs.gitlab.com/ee/ci/yaml/#hookspre_get_sources_script | hook ]]. - Note that if we use Git on the Zuul side and the `pre_get_sources_script` hook, we may be able to simply inject global Git `insteadOf` configuration which would save us redundant cloning operations and allow the GL jobs to function completely transparently with respect to the alternative Zuul Git source. 3. Continue execution normally, and report ongoing and completion status to the associated MR. Let's prove or disprove this concept.

Adopting Zuul as a primary CI system for some GitLab hosted repos will afford us valuable gating and merging functionality (see {T349872}), but we lose CI standardization and take on the burden of supporting (managing/documenting/implementing) an additional type of CI configuration. If we promote the use of `.zuul.yaml` files in individual repos, we retain CI self service. However, it's an unknown whether or not the Ansible driven configuration of Zuul is sufficient for end users, and while GitLab CI has gone through community consultation, Zuul has not. If we manage the Zuul/Ansible configuration separately from end-user repos, we lose CI self service altogether. Neither of these options seem like great outcomes. There may be an alternative, however, whereby we benefit from Zuul gating and retain CI self-service in the canonical form of `.gitlab-ci.yml` end-user configuration. 1. Integrate Zuul with GitLab for gating/merging.. 2. Manage all (or most) Zuul configuration separately from individual repos in a Zuul trusted project, including Zuul tenants, pipelines, projects, and jobs. 3. Define a standard Zuul job for all GitLab repos (e.g. `gitlab-pipeline`) that: 1. Creates a GitLab pipeline for the change (MR) using the GitLab API, passing the Zuul build UUID as a pipeline variable. 2. Exposes the speculative repo state, which includes all dependent repos, to the GitLab CI jobs via something like rsyncd/ssh/git. A shared volume may also be possible, however such an implementation would have to be possible with both k8s and Docker based GitLab runners. 3. Waits for the completion of the GL pipeline and fails/succeeds based on the result. 4. The repo-configured GitLab pipeline would in turn. 1. Configure `workflow` rules such that only Zuul triggered pipelines are run (e.g. `if: $CI_PIPELINE_SOURCE == 'api' && $ZUUL_BUILD`) 2. Sync the speculative repo state from the Zuul executor (`/var/lib/zuul/builds`) (or node `/src`) using rsync or git in either a `before_script` or `hooks:pre_get_sources_script` [[ https://docs.gitlab.com/ee/ci/yaml/#hookspre_get_sources_script | hook ]]. - Note that if we use Git on the Zuul side and the `pre_get_sources_script` hook, we may be able to simply inject global Git `insteadOf` configuration which would save us redundant cloning operations and allow the GL jobs to function completely transparently with respect to the alternative Zuul Git source. 3. Continue execution normally, and report ongoing and completion status to the associated MR. Let's prove or disprove this concept.

Adopting Zuul as a primary CI system for some GitLab hosted repos will afford us valuable gating and merging functionality (see {T349872}), but we lose CI standardization and take on the burden of supporting (managing/documenting/implementing) an additional type of CI configuration. If we promote the use of `.zuul.yaml` files in individual repos, we retain CI self service. However, it's an unknown whether or not the Ansible driven configuration of Zuul is sufficient for end users, and while GitLab CI has gone through community consultation, Zuul has not. If we manage the Zuul/Ansible configuration separately from end-user repos, we lose CI self service altogether. Neither of these options seem like great outcomes. There may be an alternative, however, whereby we benefit from Zuul gating and retain CI self-service in the canonical form of `.gitlab-ci.yml` end-user configuration. 1. Integrate Zuul with GitLab for gating/merging.. 2. Manage all (or most) Zuul configuration separately from individual repos in a Zuul trusted project, including Zuul tenants, pipelines, projects, and jobs. 3. Define a standard Zuul job for all GitLab repos (e.g. `gitlab-pipeline`) that: 1. Creates a GitLab pipeline for the change (MR) using the GitLab API, passing the Zuul build UUID as a pipeline variable. 2. Exposes the speculative repo state, which includes all dependent repos, to the GitLab CI jobs via something like rsyncd/ssh/git. A shared volume may also be possible, however such an implementation would have to be possible with both k8s and Docker based GitLab runners. 3. Waits for the completion of the GL pipeline and fails/succeeds based on the result. 4. The repo-configured GitLab pipeline would in turn. 1. Configure `workflow` rules such that only Zuul triggered pipelines are run (e.g. `if: $CI_PIPELINE_SOURCE == 'api' && $ZUUL_BUILD`) 2. Sync the speculative repo state from the Zuul executor (`/var/lib/zuul/builds`) (or node `/src`) using rsync or git in either a `before_script` or `hooks:pre_get_sources_script` [[ https://docs.gitlab.com/ee/ci/yaml/#hookspre_get_sources_script | hook ]]. - Note that if we use Git on the Zuul side and the `pre_get_sources_script` hook, we may be able to simply inject global Git `insteadOf` configuration which would save us redundant cloning operations and allow the GL jobs to function completely transparently with respect to the alternative Zuul Git source. 3. Continue execution normally, and report ongoing and completion status to the associated MR. Let's prove or disprove this concept.