Flink
SeniorDo I actually need exactly-once semantics from Flink's Kafka connector?
Exactly-once isn't free, and it isn't always necessary. The real question is whether the downstream write is naturally idempotent - that decides it, not a blanket rule about correctness.
Flink's Kafka sink can guarantee exactly-once, but that guarantee is tied to checkpoint completion - which means output latency is bounded by the checkpoint interval, not by how fast data actually flows.
What exactly-once actually costs
Flink's Kafka sink achieves exactly-once through a two-phase commit transaction tied to the checkpoint cycle - it buffers writes and only commits them to Kafka once a checkpoint completes successfully. A consumer reading with the default read_committed isolation level sees that output delayed by the full checkpoint interval, not in near real time.
When at-least-once plus idempotent writes is just as correct, for less
If the downstream write is naturally idempotent - an upsert keyed by a stable id, not an append-only append or counter increment - at-least-once delivery with no transaction overhead produces the exact same end state as exactly-once, without the checkpoint-interval latency tax. A duplicate delivery just overwrites the same key with the same value.
Where exactly-once is the only correct answer
A non-idempotent sink - incrementing a counter, appending to an audit log where a duplicate row corrupts totals - has no safe at-least-once fallback; a replayed record there is a real correctness bug, not just noise. That's the actual decision point: is the sink idempotent, not a general preference for stronger guarantees everywhere.
See what this looks like on your own pipeline.
Upload a real Spark event log, dbt run_results.json, or Flink metrics export and get
concrete, approve-before-apply recommendations back - not another rule of thumb.