Right now we are using the default dbt schema names, which are based on profiles.yaml and prefixes.
We have to change this to allow Airflow (Production) to write in the schemas we want, while dev environments (manual runs on Stat hosts) will write into personal schemas.
dbt has documentation about how to do this with configs and custom macros
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Using custom schemes | repos/data-engineering/dbt-jobs!32 | javiermonton | feature/custom-schemes | main | |
| dbt target name | repos/data-engineering/airflow-dags!2175 | javiermonton | feature/dbt-target-production | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Mayakp.wiki | T369226 [MI 1] Reporting, Data Visualization, and Data Communications | |||
| Resolved | Mayakp.wiki | T403304 [MI 1.2] Provide metric updates to staff through 2 metric-focused presentations | |||
| Resolved | Mayakp.wiki | T403305 Contributors Metrics Reporting for Staff and stakeholders | |||
| Resolved | Mayakp.wiki | T408146 [Hypothesis] WE1.5.1 Contributor metrics dashboard | |||
| Open | None | T416679 dbt DPE work | |||
| Open | amastilovic | T406764 Provide a dbt-core development environment and production setup in the data-platform | |||
| Resolved | JMonton-WMF | T423396 dbt: Use custom schemas |
javiermonton opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2175
dbt target name
javiermonton opened https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/32
Using custom schemes
javiermonton closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2175
dbt target name
javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/32
Using custom schemes