Flink

Mid-level

Why is my Flink job silently dropping late-arriving events?

A watermark is Flink's own claim that no more events older than a given time will arrive - not a measurement. A too-aggressive one closes a window before real data shows up, with no error.

Read in:
No lateness allowedLateness + side output
Roughly how much visibility each step adds into events arriving after the watermark - not a measured benchmark.

An event that arrives after its window has already fired doesn't error, doesn't log, and doesn't retry - it's just gone, and the window's result is quietly lower than the real count.

What a watermark actually asserts

A watermark is Flink's own claim - "no more events with a timestamp before T will arrive" - which is what lets a window operator decide a window is complete and fire its result. It's an assertion the job makes about the data based on a configured strategy, not a direct measurement of it, so a too-aggressive strategy can assert completeness before the real, still in-flight data actually shows up.

Where late events actually go

By default, an event arriving after its window has already fired is simply dropped - no error, no log line. allowedLateness keeps a window's state open for a configured extra period specifically to fold late events like this one back in, updating the result instead of discarding it.

# keeps the window open 5 extra minutes for late data .window(...).allowedLateness(Time.minutes(5))

The side-output you probably don't have

Even with allowedLateness set, an event arriving after that extended window finally closes is still dropped. Routing it to a .sideOutputLateData() stream instead - even just to count and alert on the volume - is often the only way to actually find out how much of this is happening, since nothing about the main pipeline's output will tell you.

See what this looks like on your own pipeline.

Upload a real Spark event log, dbt run_results.json, or Flink metrics export and get concrete, approve-before-apply recommendations back - not another rule of thumb.