Adam Richardson
06/26/2025, 3:27 AMorg.apache.flink.api.connector.sink.{Writer,Committer,GlobalCommitter,Sink,...}). I'm observing strange behavior where consumer group latency on Kafka is close to zero after a checkpoint, but the data in the output Delta table lags significantly. I can see in the TM logs that Delta commits are happening quickly and reliably after each checkpoint, but recent data is still missing.
After some digging I found a reference in an unrelated issue on Flink: "it is expected to global committer architecture lag one commit behind in reference to rest of the pipeline". This lines up very well with the symptoms I'm seeing -- my app checkpoints every 5m, and I'm seeing latency in the output table fluctuating from 5m (immediately after a checkpoint) to 10m (immediately before the next checkpoint).
My questions:
• Is it expected behavior that the committed data consistently lags behind by one checkpoint? Are there any more authoritative docs on this behavior?
• Is there any workaround (config or code changes) to avoid/fix this behavior?
• Does the v2 sink API have this problem?
Thank you!