How do I get alerted before a pipeline problem becomes expensive?

Most pipeline incidents don't start as incidents — they start as a slow trend that nobody was watching until it crossed some threshold and became an emergency. You don't need a full observability platform to catch the trend early; you need to be watching the right small number of signals.

The handful of metrics that actually predict trouble

• GC time as a % of task time (Spark) — climbing steadily over consecutive runs predicts an OOM before it happens, often weeks ahead of the actual failure.
• Checkpoint duration trend (Flink) — growing checkpoint time predicts eventual checkpoint timeouts and the job restarts that follow, well before the first timeout actually occurs.
• Shuffle spill bytes (Spark) — a slow creep here means your shuffle partition sizing hasn't kept up with data growth, well before it becomes a multi-hour runtime regression.
• Elapsed time vs. row count, per model (dbt) — if this ratio is worsening run over run, a model is scaling worse than linearly with your data growth, and will keep getting worse.

Trend beats threshold

A fixed alert threshold ("page me if runtime exceeds 30 minutes") only fires after the problem is already real. Comparing each run against a rolling baseline from recent runs catches the trend while it's still small and cheap to fix — the same metric, used proactively instead of reactively, by asking "is this worse than the last N runs" instead of "did this cross an arbitrary line."

You don't need every metric — you need the right handful, tracked consistently

The temptation with observability is to instrument everything. The more durable approach for a small team: pick the handful of metrics above that are specific to your pipeline type, track them on every run (not just when something feels wrong), and treat a bad trend in any of them as worth a look — long before it's worth a page.