Provide a dbt-core development environment and production setup in the data-platform
Open, HighPublic
Actions

Assigned To

Authored By

	BTullis
	Oct 8 2025, 4:57 PM

Description

In support of "WE1.5.1 Contributor metrics dashboard" we want to enable dbt-core on the data platform for metrics and analyst teams and Data-Engineering.

A metrics engineer or analyst should be able to setup a development environment (e.g. on a stats machine) to develop and test new dbt based workflows, and publish them to run periodically scheduled on Airflow.

We will also want to connect it to various adapters, including: dbt-spark (preferred)

This is a parent ticket to track the various elements of work that we need to make this happen.

Related Objects
Search...

Status	Assigned	Task
Open	Mayakp.wiki	T369226 [MI 1] Reporting, Data Visualization, and Data Communications
In Progress	Mayakp.wiki	T403304 [MI 1.2] Provide metric updates to staff through 2 metric-focused presentations
In Progress	Mayakp.wiki	T403305 Contributors Metrics Reporting for Staff and stakeholders
Open	Mayakp.wiki	T408146 [Hypothesis] WE1.5.1 Contributor metrics dashboard
Open	amastilovic	T406764 Provide a dbt-core development environment and production setup in the data-platform
Resolved	JMonton-WMF	T406636 Create a dbt Docker container
Resolved	Ahoelzl	T406634 Set up a working, usable dbt installation on stat boxes
Resolved	BTullis	T406766 Add dbt related packages to conda-analytics
Duplicate	None	T406767 Add dbt related packages to conda-analytics
Resolved	None	T406765 Create a new gitlab repository for use with dbt
Resolved	JMonton-WMF	T407322 Create dbt folder structure
Resolved	JMonton-WMF	T408687 Create example dbt models using Iceberg
Resolved	JMonton-WMF	T409054 Explore a local dbt environment setup (independent from Conda)
Resolved	JMonton-WMF	T409099 Iceberg Merge strategies with dbt
Open	JMonton-WMF	T410268 Run dbt from Airflow
Open	BTullis	T410017 Provide a Spark production access for dbt with Airflow
Open	BTullis	T413977 Deploy Kyuubi to enable dynamic spark-sql clusters in dse-k8s

Event Timeline

BTullis created this task.Oct 8 2025, 4:57 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2025, 4:57 PM

BTullis triaged this task as High priority.Oct 8 2025, 4:58 PM

BTullis added subtasks: T406636: Create a dbt Docker container, T406634: Set up a working, usable dbt installation on stat boxes.

Ahoelzl moved this task from Incoming (new tickets) to Q1 FY25/26 July 1st - September 30th on the Data-Engineering board.Oct 8 2025, 5:07 PM

Ahoelzl edited projects, added Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed Data-Engineering.

BTullis added subscribers: Ahoelzl, JAllemandou, Ottomata.Oct 8 2025, 5:07 PM

Ottomata assigned this task to amastilovic.Oct 8 2025, 5:07 PM

GGoncalves-WMF subscribed.Oct 8 2025, 5:14 PM

Ahoelzl edited projects, added Data-Engineering (Q2 FY25/26 October 1st - December 31th); removed Data-Engineering (Q1 FY25/26 July 1st - September 30th).Oct 9 2025, 12:13 AM

Mayakp.wiki subscribed.Oct 15 2025, 4:02 PM

Gehel moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.Oct 16 2025, 8:11 AM

Mayakp.wiki added a project: Movement-Insights.Oct 17 2025, 5:46 PM

Mayakp.wiki moved this task from Incoming to Watching on the Movement-Insights board.

GGoncalves-WMF renamed this task from Exlore the use of dbt-core and appropriate adapters in the data-platform environment to Explore the use of dbt-core and appropriate adapters in the data-platform environment.Oct 21 2025, 1:49 PM

Pasting here the slack comment I wrote yesterday:

Good progress made this evening: DBT runs with spark in session mode on the test cluster, querying data and writing data. There is lot more learning to be done on how to make DBT do what we expect it to do (partition creation, iceberg dates etc), but at least the basic infra is working!

xcollazo moved this task from Next Up to Done on the Data-Engineering (Q2 FY25/26 October 1st - December 31th) board.Nov 3 2025, 4:09 PM

Ahoelzl closed this task as Resolved.Nov 18 2025, 10:25 PM

Ahoelzl closed subtask T409099: Iceberg Merge strategies with dbt as Resolved.

Ahoelzl closed subtask T409054: Explore a local dbt environment setup (independent from Conda) as Resolved.

Ahoelzl closed subtask T406636: Create a dbt Docker container as Resolved.

Ahoelzl closed subtask T408687: Create example dbt models using Iceberg as Resolved.

Ahoelzl closed subtask T406765: Create a new gitlab repository for use with dbt as Resolved.Nov 18 2025, 11:07 PM

Ahoelzl renamed this task from Explore the use of dbt-core and appropriate adapters in the data-platform environment to Provide dbt-core and appropriate adapters in the data-platform environment.Nov 18 2025, 11:25 PM

Ahoelzl reopened this task as Open.

Ahoelzl updated the task description. (Show Details)

Ahoelzl moved this task from Done to In progress on the Data-Engineering (Q2 FY25/26 October 1st - December 31th) board.

Ahoelzl added a subtask: T410017: Provide a Spark production access for dbt with Airflow.

Ahoelzl added a parent task: T408146: [Hypothesis] WE1.5.1 Contributor metrics dashboard.

Ahoelzl renamed this task from Provide dbt-core and appropriate adapters in the data-platform environment to Provide a dbt-core development environment and production setup in the data-platform.Nov 18 2025, 11:29 PM

Ahoelzl updated the task description. (Show Details)