Flink

Senior

Do I actually need exactly-once semantics from Flink's Kafka connector?

Exactly-once isn't free, and it isn't always necessary. The real question is whether the downstream write is naturally idempotent - that decides it, not a blanket rule about correctness.

Read in:
At-least-once Exactly-once (2PC) Checkpoint-bound delay
Roughly how relative output latency scales with the delivery guarantee chosen - not a measured benchmark.

Flink's Kafka sink can guarantee exactly-once, but that guarantee is tied to checkpoint completion - which means output latency is bounded by the checkpoint interval, not by how fast data actually flows.

What exactly-once actually costs

Flink's Kafka sink achieves exactly-once through a two-phase commit transaction tied to the checkpoint cycle - it buffers writes and only commits them to Kafka once a checkpoint completes successfully. A consumer reading with the default read_committed isolation level sees that output delayed by the full checkpoint interval, not in near real time.

# consumer-side setting that actually enforces the guarantee isolation.level=read_committed

When at-least-once plus idempotent writes is just as correct, for less

If the downstream write is naturally idempotent - an upsert keyed by a stable id, not an append-only append or counter increment - at-least-once delivery with no transaction overhead produces the exact same end state as exactly-once, without the checkpoint-interval latency tax. A duplicate delivery just overwrites the same key with the same value.

Where exactly-once is the only correct answer

A non-idempotent sink - incrementing a counter, appending to an audit log where a duplicate row corrupts totals - has no safe at-least-once fallback; a replayed record there is a real correctness bug, not just noise. That's the actual decision point: is the sink idempotent, not a general preference for stronger guarantees everywhere.

See what this looks like on your own pipeline.

Upload a real Spark event log, dbt run_results.json, or Flink metrics export and get concrete, approve-before-apply recommendations back - not another rule of thumb.