Page MenuHomePhabricator
Create Task
Maniphest T379545

Update the DAGs on the platform_eng airflow instance to use miniforge instead of condaforge and mambaforge
Open, In Progress, Needs TriagePublic
Actions

Description

As per the parent ticket, there are five DAGs running on the platform_eng airflow instance which use WMF Data Workflow Utils to build a runtime conda environment, based on miniconda.

Due to upcoming licence changes in the Anaconda project, we wish to ensure that all of these environments are using the latest version of the workflow utils conda pipeline, in which we switch from miniconda to miniforge.

Please would you take steps to upgrade these DAGs and deploy them, when convenient?

It should just be a case of ensuring the gitlab-ci.yml file references v0.19.0 of repos/data-engineering/workflow_utils as stated here:
https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide/Python_Job_Repos#GitLab_CI_setup

Some of the jobs also install mambaforge manually.
This should no longer be required and it would be best to remove this for the same reasons.

There is a reference GitLab MR here, which you may find useful:
https://gitlab.wikimedia.org/repos/data-engineering/example-job-project/-/merge_requests/36/

Related Objects
Search...

Event Timeline

BTullis created this task.Nov 11 2024, 11:29 AM
Restricted Application added a project: Structured-Data-Backlog. · View Herald TranscriptNov 11 2024, 11:29 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
BTullis added a parent task: T372417: Switch from miniconda to miniforge.Nov 11 2024, 11:30 AM
mfossati updated the task description. (Show Details)Mon, Nov 18, 3:16 PM
AUgolnikova-WMF moved this task from Triage to Current Work on the Structured-Data-Backlog board.Mon, Nov 18, 5:26 PM
AUgolnikova-WMF edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.
AUgolnikova-WMF moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.
AUgolnikova-WMF moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Mon, Dec 2, 5:21 PM
mfossati changed the task status from Open to In Progress.Fri, Dec 6, 4:49 PM
mfossati claimed this task.
mfossati moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.
mfossati updated the task description. (Show Details)Fri, Dec 6, 5:07 PM
CodeReviewBot added a project: Patch-For-Review.Fri, Dec 6, 5:57 PM

mfossati opened https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/45

Bump workflow utils & all dependencies

CodeReviewBot added a comment.Fri, Dec 6, 6:22 PM

mfossati opened https://gitlab.wikimedia.org/repos/structured-data/seal/-/merge_requests/3

Bump workflow utils & most dependencies

CodeReviewBot added a comment.Mon, Dec 9, 4:14 PM

mfossati opened https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/38

update CI

mfossati moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Tue, Dec 10, 1:26 PM
mfossati mentioned this in T368931: Unify CI jobs.Tue, Dec 10, 1:29 PM
mfossati updated the task description. (Show Details)
mfossati added a comment.Tue, Dec 10, 1:39 PM

@Htriedman I'll let you update https://gitlab.wikimedia.org/repos/security/differential-privacy/-/blob/main/.gitlab-ci.yml, CC @BTullis .

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits