What is the issue?
For dbt jobs when we enable data tests for the job, with runs_tests: true in the config YAML, it only run the tests but not the actual job.
In the Airflow DAG that's scheduled, the actual job tasks runs as well, but strangely, if we observe the logs, it is running the tests instead of the main job when the main job is executed.
I observed this in a couple of instances so far: dbt_responsible_traffic_pathways and known_clients_aggregate_daily -- both have data tests enabled.
For the dbt_responsible_traffic_pathways: should start with
10:31:14 1 of 2 OK created sql incremental model ... 10:31:14 2 of 2 START sql incremental model ... 10:31:16 2 of 2 OK created sql incremental model ...
instead from the logs we can observe, it starts with tests directly for the main job as well.
00:01:17 1 of 22 01:17 1 of 22 START test accepted_values_responsible_traffic_pathways_daily_http_outcome__success__ratelimited__denied__other 00:01:43 1 of 22 PASS accepted_values_responsible_traffic_pathways_daily_http_outcome__success__ratelimited__denied__other [PASS in 26.08s] 00:01:43 2 of 22 START test accepted_values_responsible_traffic_pathways_daily_traffic_segment__api_gateway__api_non_gateway__media__pageviews__wme_enterprise__dumps__other_text 00:01:52 2 of 22 PASS accepted_values_responsible_traffic_pathways_daily_traffic_segment__api_gateway__api_non_gateway__media__pageviews__wme_enterprise__dumps__other_text [PASS in 8.81s] ... 00:02:05 Finished running 22 data tests in 0 hours 1 minutes and 5.69 seconds (65.69s). ... 00:02:05 Done. PASS=22 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=22
Also, the main job should take about ~30 min to be complete, whereas in this case as it is actually running tests, it completed within a couple of minutes.
What should have happened instead?
- The main jobs should run the incremental model, followed by data tests.
This currently affects the following jobs