Maniphest T315475

Convert to pure Docker the gitlab CI pipeline to build debianized conda
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Antoine_Quhen
	Aug 17 2022, 6:50 PM

Project Tags

Referenced Files

None

Subscribers

Description

We currently maintain 2 processes to build the Debian conda env:

some bash scripts in workflow utils defined within YAML files to create the production version of the deb file
+ a multistep Dockerfile for dev+debug purpose (faster)

It would be great to have only 1 Dockerfile for both purposes.

Event Timeline

Antoine_Quhen created this task.Aug 17 2022, 6:50 PM

Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptAug 17 2022, 6:50 PM

Antoine_Quhen added a parent task: T295072: Install spark3 in analytics clusters.Aug 17 2022, 6:50 PM

• EChetty edited projects, added: Data-Engineering-Planning; removed: Data-Engineering.Sep 6 2022, 10:40 AM

• EChetty moved this task from Backlog to Pipelines on the Data-Engineering-Planning board.Sep 6 2022, 10:46 AM

We have the first version here:

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/c3641457551c6686695ee4f35374e5b7954f7920/debian/Dockerfile The Docerkfile is an important tool allowing me to develop and test the deb pkg locally on my machine.
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/c3641457551c6686695ee4f35374e5b7954f7920/.gitlab-ci.yml#L58-90

but the docker.latest image is not allowed in docker-ci.

So, from the Dockefile, I rewrote the Dockerfile into a YAML bash script:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/c3641457551c6686695ee4f35374e5b7954f7920/.gitlab-ci.yml#L110-160

As a result, it's an error-prone process, where the development environment is far from the build environment.

BTullis subscribed.Oct 13 2022, 9:36 AM

xcollazo closed this task as a duplicate of T321736: Modify conda-analytics CI pipeline to use a custom gitlab runner that can run docker.Nov 2 2022, 8:17 PM

Whoops! I though this task referred to conda-analytics but in fact is for airflow-dags. Sorry, reopening...

JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:59 PM

JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:53 PM

• lbowmaker edited projects, added: Data-Engineering; removed: Data Engineering and Event Platform Team.Nov 10 2023, 2:49 PM

nshahquinn-wmf removed a parent task: T295072: Install spark3 in analytics clusters.Feb 9 2024, 9:22 PM

• lbowmaker moved this task from Incoming (new tickets) to Tag with Icebox on the Data-Engineering board.Feb 16 2024, 8:52 PM

• VirginiaPoundstone edited projects, added: Data-Engineering-Icebox-DEPRECATED; removed: Data-Engineering.Jan 6 2025, 9:19 PM

Ottomata edited projects, added: Data-Engineering, Data-Engineering-Icebox; removed: Data-Engineering-Icebox-DEPRECATED.Jan 10 2025, 2:38 AM