dbt
SeniorHow do I stop every CI run from rebuilding my entire dbt project?
dbt builds everything by default. State-based selection is the only thing that makes CI cost scale with how much actually changed, not with how big the project has gotten.
A dbt project with 40 models and one with 400 run the exact same dbt build command - and pay very different CI bills for it, unless something is actually telling dbt what changed.
What state-based selection actually compares
dbt writes a manifest.json after every successful run, which fully describes every model's compiled SQL and its place in the DAG. --select state:modified+ diffs the current manifest against a stored one you point it at, and builds only models whose compiled SQL actually changed, plus everything downstream of them.
Why this quietly never kicks in
The single most common reason state selection does nothing: CI never persisted or fetched the previous run's manifest, so --state has nothing to diff against. dbt doesn't error on this - it just can't find modified models relative to nothing, and the build quietly does everything. Fixing it means treating manifest.json as a CI artifact that gets uploaded after every successful production run and pulled down at the start of the next one.
The other silent killer: full-refresh habits
A --full-refresh flag kept in the CI command "to be safe" rebuilds every incremental model from scratch regardless of what state selection found - it ignores the diff entirely for those models. That default is usually a leftover from debugging one broken incremental model once, never removed. Reserve --full-refresh for an explicit, deliberate run against one model, not a standing CI flag.
The tell: if CI build time scales with total project size instead of PR diff size, state selection isn't actually running - check for a missing --state manifest first, a stray --full-refresh second.
See what this looks like on your own pipeline.
Upload a real Spark event log, dbt run_results.json, or Flink metrics export and get
concrete, approve-before-apply recommendations back - not another rule of thumb.