How do I pick the right Structured Streaming trigger interval and micro-batch size?

A shorter trigger interval sounds like it should always mean lower latency. Past a point, it just means more batches spent on bookkeeping instead of data.

What the trigger interval actually controls

Trigger.ProcessingTime batches whatever new data arrived within that window into one micro-batch, then plans, executes, commits, and checkpoints it as a unit. A shorter interval means lower end-to-end latency per record, but every one of those steps - planning, commit, checkpoint - has a roughly fixed cost regardless of how much data is actually in the batch.

Why shrinking the interval can make throughput worse

If that fixed per-batch overhead is close to the interval itself, the job spends more wall-clock time on batch bookkeeping than on actual processing. It shows up as batches that "complete" almost instantly in the Spark UI while a growing input backlog queues up behind them - the pipeline looks healthy per-batch and is falling behind overall.

Trigger.AvailableNow for bounded backfills

A one-off backfill or catch-up run isn't a standing streaming job and shouldn't use the standard interval trigger at all - Trigger.AvailableNow processes exactly what's currently available, in batches, then stops on its own. Using the wrong trigger type here is a common source of a "backfill" job that never actually finishes.