Page MenuHomePhabricator

Migrate custom gitlab runner that runs Dockerfiles to releng's new production infra
Open, Needs TriagePublic

Description

In a recent Slack conversation with the releng team, we discussed how to get rid of a custom gitlab runner that was put in place on T321736. This custom runner allows us to run 'docker besides docker', with the final goal of building and running a Dockerfile that generates an production artifact.

Instead of running 'docker besides docker', we live coded a proof of concept on using buildkit to build and extract the artifact without the need of 'docker besides docker'.

In this task we should:

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Add project repos/data-engineering/conda-analyticsrepos/releng/gitlab-trusted-runner!27aquT326570_add_data_engineering_conda-analytics_projectmain
Add repos/data-engineering/airflow-dags to projectsrepos/releng/gitlab-trusted-runner!26aquT326570_add_data_engineering_airflow_dags_projectmain
Customize query in GitLab

Event Timeline

@Antoine_Quhen Copying questions from MR 417 here so that we can discuss separately:

Should we merge those CI pipelines changes to make it the standard in workflow-utils?

Can you elaborate?

Should we update conda-analytics CI?

Yes please. If we are to get rid of the custom gitlab runner completely, we also need similar improvements in conda-analytics.

Should we merge those CI pipeline changes to make it the standard in workflow utils?

We have three new pipelines in this MR:

  • Build image
  • Run tests
  • Build & publish artifacts into the registry

We could extract most of them in workflow utils to be reused. Yet, I feel it's an added complexity, for not an obvious gain of code deduplication.

Should we update conda-analytics CI?

What I really mean is, should we do it now, at the same time as the current MR (which was mainly about linting)

Should we merge those CI pipeline changes to make it the standard in workflow utils?

We have three new pipelines in this MR:

  • Build image
  • Run tests
  • Build & publish artifacts into the registry

We could extract most of them in workflow utils to be reused. Yet, I feel it's an added complexity, for not an obvious gain of code deduplication.

Ah, got it. I'd rather keep them separate so that projects can evolve independently. I think this would be a similar decision as when we chose for their Dockerfiles to also be decoupled.

Should we update conda-analytics CI?

What I really mean is, should we do it now, at the same time as the current MR (which was mainly about linting)

I think they are separate pieces of work, so no need to do conda-analytics right away.