There are several ways we could use to run dbt from Airflow, some might more complicated or give more features. If there is a quick and easy way of running dbt, we could start by that and move to a better one in the future.
The options we are considering for now are:
We also need to setup variables to be passed from Airflow to dbt
Task is done if:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| feature/dbt_monthly_dag | repos/data-engineering/airflow-dags!1964 | javiermonton | feature/dbt_monthly_dag | T415194_Create_DBT_Skein_Operator | |
| Deploy to HDFS | repos/data-engineering/dbt-jobs!15 | javiermonton | feature/deploy-to-hdfs | main | |
| Add repos/data-engineering/dbt-jobs to the trusted runners | repos/releng/gitlab-trusted-runner!136 | btullis | add_dbt_jobs | main | |
| Publish docker image. | repos/data-engineering/dbt-jobs!9 | javiermonton | feature/publish-docker | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Mayakp.wiki | T369226 [MI 1] Reporting, Data Visualization, and Data Communications | |||
| Resolved | Mayakp.wiki | T403304 [MI 1.2] Provide metric updates to staff through 2 metric-focused presentations | |||
| Resolved | Mayakp.wiki | T403305 Contributors Metrics Reporting for Staff and stakeholders | |||
| Resolved | Mayakp.wiki | T408146 [Hypothesis] WE1.5.1 Contributor metrics dashboard | |||
| Open | None | T416679 dbt DPE work | |||
| Open | amastilovic | T406764 Provide a dbt-core development environment and production setup in the data-platform | |||
| Resolved | amastilovic | T410268 Run dbt from Airflow | |||
| Resolved | amastilovic | T414784 Test the dbt+skein approach to running dbt Spark jobs in K8s | |||
| Resolved | amastilovic | T415194 Create a `DbtSkeinOperator` in the Airflow `wmf_airflow_common` library | |||
| Resolved | JMonton-WMF | T415338 Update Blunderbuss-bugler |
btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/136
Add data-engineering/refinery to the trusted runners
bd808 updated https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/136
Add repos/data-engineering/dbt-jobs to the trusted runners
bd808 merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/136
Add repos/data-engineering/dbt-jobs to the trusted runners
javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/9
Publish docker image.
javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/15
Deploy to HDFS
javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/15
Deploy to HDFS
javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1964
Draft: feature/dbt_monthly_dag
amastilovic merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1964
feature/dbt_monthly_dag