dbt
Mid-levelHow do I speed up dbt models and cut Snowflake/BigQuery compute cost?
In a dbt project, run time and warehouse bill are almost the same metric wearing two names — fixing one usually fixes the other.
Unlike a Spark cluster you provision yourself, a dbt project running on Snowflake/BigQuery bills you for warehouse compute time — so a slow model isn't just an annoyance, it's a direct cost. That makes the diagnosis list shorter and more actionable than it might seem.
Start with elapsed_time vs. execution_time, per model
dbt's own run_results.json (written after every dbt run/build)
already has both: total elapsed time for the invocation, and per-model execution time. A model whose
execution time is small relative to total elapsed time isn't the problem — something upstream (warehouse
queueing, dependency ordering) is. A model whose execution time actually dominates the run is where to
look first.
Materialization choice is usually the biggest single lever
A model materialized as a view that gets queried by many downstream models effectively
re-executes its own logic every single time it's referenced — turning what should be a one-time cost into
a repeated one. Switching a frequently-referenced, expensive transformation to table or
incremental often cuts total warehouse compute dramatically, at the cost of some storage and
a bit of staleness.
Incremental models done wrong cost more than full refreshes
An incremental model with a poorly chosen unique_key or an is_incremental()
filter that doesn't actually narrow the scan (scanning the full source table every run "incrementally")
gets you all of the complexity of incremental materialization with none of the savings. Worth explicitly
checking: does the incremental run actually scan less data than a full refresh would, per the warehouse's
own query profile?
Parallelism (dbt's threads config) has a ceiling
More threads means more models running concurrently against the warehouse — up to the point where they're all competing for the same warehouse compute slots, at which point more threads just means more queueing, not more actual throughput. This ceiling is warehouse-size-dependent, so a value copied from another project's config isn't necessarily right for yours.
What opti-pipe reads from a dbt run: total elapsed time and per-model execution time directly
out of run_results.json — never your SQL, model names, or file paths — to flag which models
are actually driving the run's cost, the same "look at the real numbers, not a guess" approach as the
Spark rules above.
See what this looks like on your own pipeline.
Upload a real Spark event log, dbt run_results.json, or Flink metrics export and get
concrete, approve-before-apply recommendations back — not another rule of thumb.