How do I configure Spark's dynamic allocation from scratch, not just tune it after the fact?

spark.dynamicAllocation.enabled=true alone doesn't do anything on most cluster managers. Dynamic allocation needs an external shuffle service configured and running first - without it, executors can't safely be removed mid-job, so Spark just... doesn't remove them.

The prerequisite nobody mentions first

Removing an executor mid-job is only safe if the shuffle data it holds stays available to other executors after it's gone - that's what the external shuffle service does, running independently of any single executor's lifecycle. Without it enabled, Spark accepts dynamicAllocation.enabled=true without error and then simply never scales down, because doing so would risk losing shuffle data. On YARN and Kubernetes this is a separate service that has to be deployed and pointed at from Spark's config - it's not automatic just because dynamic allocation is turned on.

Setting real bounds, not just a ceiling

Leaving minExecutors at its default of 0 means a job can scale all the way down to zero executors during a lull and then pay full cluster-manager allocation latency to scale back up - fine for a batch job with slack, costly for anything latency-sensitive. maxExecutors is the more familiar knob (it's the cost ceiling), but it's only half the picture without a matching floor:

Idle timeout, and where this still needs judgment

spark.dynamicAllocation.executorIdleTimeout (default 60s) controls how long an executor sits idle before being released - too short and a job with bursty, uneven stages thrashes executors up and down constantly, adding allocation latency on every burst; too long and you're paying for idle capacity between bursts. There's no formula for the right value - it depends on how spiky the workload actually is, which is exactly the kind of pattern worth looking at in a real run's timeline before picking a number.