Speculative execution in Spark: safety net or hidden cost multiplier?

When speculation is enabled, Spark watches for tasks running significantly slower than the median in their stage and launches a duplicate attempt elsewhere, keeping whichever finishes first.

The case where it's exactly right

A single task stuck at 10x the median duration while every other task in the stage finishes normally is the textbook case: a flaky node, a noisy neighbor on shared infrastructure, a transient disk or network hiccup. Speculation launches a duplicate, the duplicate finishes on a healthy node, and the job recovers without anyone paging anyone - exactly the safety net it's designed to be.

The case where it makes things worse

Data skew produces the same surface symptom - one task much slower than the rest - for a completely different reason: that task just has more data to process. Speculation launches a duplicate of a task that was never going to finish quickly no matter which node runs it, so you pay for two slow attempts instead of one, with no improvement in the actual completion time. The skewed task still finishes last either way.

Telling them apart before you flip the switch

Check task input size, not just duration, for the slow task versus its peers in the same stage. Roughly even input size with one wildly slower task points to an infrastructure issue speculation will actually help with. A slow task with visibly more input data (from "Input Metrics" or shuffle-read bytes in the event log) is skew - the fix there is repartitioning or salting the join key, not speculation, which just doubles the cost of a task that was always going to be the long pole.