Hi all. I have a few questions about the legacy tw...
# troubleshooting
a
Hi all. I have a few questions about the legacy two-phase commit sink. My application reads data from Kafka and writes it to a Delta Lake via the open-source Delta sink, which is implemented as a 2PC sink using the legacy APIs (
org.apache.flink.api.connector.sink.{Writer,Committer,GlobalCommitter,Sink,...}
). I'm observing strange behavior where consumer group latency on Kafka is close to zero after a checkpoint, but the data in the output Delta table lags significantly. I can see in the TM logs that Delta commits are happening quickly and reliably after each checkpoint, but recent data is still missing. After some digging I found a reference in an unrelated issue on Flink: "it is expected to global committer architecture lag one commit behind in reference to rest of the pipeline". This lines up very well with the symptoms I'm seeing -- my app checkpoints every 5m, and I'm seeing latency in the output table fluctuating from 5m (immediately after a checkpoint) to 10m (immediately before the next checkpoint). My questions: • Is it expected behavior that the committed data consistently lags behind by one checkpoint? Are there any more authoritative docs on this behavior? • Is there any workaround (config or code changes) to avoid/fix this behavior? • Does the v2 sink API have this problem? Thank you!