We currently have sqlfluff lint as a CI check for new MRs in the dbt-jobs repository. More than validating SQL syntax and style, it also indirectly runs dbt compile, which catches integrity errors in the dbt DAG (e.g., referencing a model that doesn't exist).
This check runs again a fake database, because dbt compile requires a working connection. Since this database is always empty, dbt never triggers the is_incremental() path in models, because it thinks it's doing an initial full run.
Since dbt jobs most often run in incremental mode in production, we should figure out how to get those code blocks covered in CI, both for lint and (especially) DAG integrity.
Acceptance criteria
- dbt compile (as part of lint, or a separate step) runs as a CI check in incremental mode.
- SQLFluff can also lint code in is_incremental() blocks.