It seems like it's mostly models in infrastructure-metrics written by @KCVelaga_WMF :) KC, would you mind chiming in on this? Do these models need such a partitioning layout?

The new schedule is live on Airflow.

OK here's the proposed config file layout. It intentionally reuses some existing components from datasets.yaml, and I believe it is flexible enough to facilitate easy creation of rules that will cover all of our use cases.

@xcollazo correct, we will need a dbt_analytics user and user group, with Kerberos ticketing.

Just thought of something important:

Thanks for the comments, @xcollazo and @JAllemandou !

@JAllemandou @GGoncalves-WMF can we further discuss this please, since I believe there are some unresolved issues with this solution.

I believe this ticket is now resolved.

Backfill completed, but I had to break it into two pieces: 2020-01-01 to 2023-01-01, and the second run was 2023-01-01 to 2026-07-20.

Task completed.

Incorrect data rows were deleted.

Style guide updated: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/dbt/dbt-jobs_repository_style_guide#Choosing_a_scheduling_group_start_date

@Gehel as per @Ottomata , Data Engineering doesn't have any Python projects in Archiva. Can this ticket be resolved?

I believe datahub-cli has already been migrated to GitLab, publishing its artifacts to the GitLab Package Registry: https://gitlab.wikimedia.org/repos/data-engineering/datahub-cli

1. Automated retention pruning for dbt staging/intermediate Iceberg tables

Ideally, if a model owner forgets to specify start_date, this would be caught at MR review time or, failing that, cause a failure in production only for that model's group. If we go that way, would that be feasible @amastilovic ?

@nshahquinn-wmf would it be possible for the dbt model to be rewritten so that it accepts the ordinal number of the week to process within a given month? Or a "sub-date range" within the month range? If something like this is possible then we could schedule four separate runs to occur each month, that would cover the previous month's interval. Airflow scheduling would most likely be trivial, as would be the backfills.

@jwang I'm not sure I understand which rows should be deleted? These dbt jobs are usually incremental which effectively means that re-running the month will replace the previous results for that month.

Backfill completed.

This backfill is now completed.

Running it now.

OK I've created the Postgres schemas and backfilled all tables except for traffic_pathways_responsible_daily - that one is actually named responsible_traffic_pathways_daily in Hive so it required another MR which was just merged - it should show up for tomorrow's Airflow run.

I've scheduled the Datalake-2-Postgres transfers in Airflow, but ran into the following error while trying to backfill:

By the way, since this particular run (2026-08) was already in Airflow, all you need to do to re-run it is go to https://airflow.wikimedia.org/dags/dbt_fy_2026_2027_monthly/grid find the task run, select the square and click the "Clear task" blue button in the upper right corner.

I've re-run the 2026-08 job and here are the results of the verification query:

I guess we could:
Increment the SLA?

We can do this, I'll submit an MR

Revise the heavy queries to either optimize them or adjust their cluster resources?

I'll go and review the performance of these tasks, if there's anything to adjust I'll submit an MR

Increase the amount of tasks that the DAG can run at the same time?

I intentionally set the max number of tasks running to 1 because having the same Airflow DAG run two or more dbt run commands at the same time was causing dbt run to fail, I don't think we should mess with this.

I think this can be achieved using the following Jinja template functions that are already provided in our Airflow deployment (mwh_snapshot in this example):

@nshahquinn-wmf thanks for some context and real-world examples! Although this is not about the default start_date but the earliest allowed date, I think your comment is still relevant.

@pfischer I think this ticket can be close/marked done.

1. Proposal: two-stage dbt compile (no dbt build/run/test anywhere)

This should now be fixed - it took a little while longer because the fix touched on many different teams' production stuff so we had to communicate the change to everyone and get their approvals.

I've identified the issue and created the MR to fix it. The root cause is the fact that Iceberg dataset sensor construction did not heed the grain discrepancy between the caller DAG and the dataset. Once the MR linked above is merged, I'll go ahead and update all Iceberg datasets in datasets.yaml with the cadence/grain information.

Task completed!

Backfill completed.

Done.

Done.

Wikitech docs updated.

Airflow code and YAML configs updated.

Awesome - thank you @EBernhardson !

@GGoncalves-WMF I think this ticket can be resolved/closed now.

Thank you for the update @atsuko