dbt

Senior

How do I stop every CI run from rebuilding my entire dbt project?

dbt builds everything by default. State-based selection is the only thing that makes CI cost scale with how much actually changed, not with how big the project has gotten.

Read in:
Full rebuild state:modified+ Changed models only
Roughly how CI build scope narrows as state-based selection is added - not a measured benchmark.

A dbt project with 40 models and one with 400 run the exact same dbt build command - and pay very different CI bills for it, unless something is actually telling dbt what changed.

What state-based selection actually compares

dbt writes a manifest.json after every successful run, which fully describes every model's compiled SQL and its place in the DAG. --select state:modified+ diffs the current manifest against a stored one you point it at, and builds only models whose compiled SQL actually changed, plus everything downstream of them.

# --state points at yesterday's manifest, not today's dbt build --select state:modified+ --state ./prod-manifest

Why this quietly never kicks in

The single most common reason state selection does nothing: CI never persisted or fetched the previous run's manifest, so --state has nothing to diff against. dbt doesn't error on this - it just can't find modified models relative to nothing, and the build quietly does everything. Fixing it means treating manifest.json as a CI artifact that gets uploaded after every successful production run and pulled down at the start of the next one.

The other silent killer: full-refresh habits

A --full-refresh flag kept in the CI command "to be safe" rebuilds every incremental model from scratch regardless of what state selection found - it ignores the diff entirely for those models. That default is usually a leftover from debugging one broken incremental model once, never removed. Reserve --full-refresh for an explicit, deliberate run against one model, not a standing CI flag.

The tell: if CI build time scales with total project size instead of PR diff size, state selection isn't actually running - check for a missing --state manifest first, a stray --full-refresh second.

See what this looks like on your own pipeline.

Upload a real Spark event log, dbt run_results.json, or Flink metrics export and get concrete, approve-before-apply recommendations back - not another rule of thumb.