Hello, I am trying to use Flink to stream data to Iceberg table. For testing purpose, I’m using a custom input that continuously generate dummy data. I have observed that if I make my stream finite, says 10 messages, then all 10 are saved to Iceberg and the job stops. No problem. However, if I make my stream infinite, then the job keeps running (as expected) but no row is inserted to Iceberg table. UNLESS I enable checkpoint with

env.enableCheckpointing(1000);

, then I see rows inserted into the table again. Why is that? How does enabling checkpoint makes Flink actually insert to iceberg table?

This is a specific logic of Iceberg connector - it commits data to tables on checkpoint events. However, this is not mentioned clearly in Iceberg docs, only reference is in metric description:

Iceberg commit happened after successful Flink checkpoint in the

notifyCheckpointComplete

callback.

https://iceberg.apache.org/docs/1.2.0/flink/#netrics:~:text=Iceberg%20commit%20happened%20after%20successful%20Flink%20checkpoint%20in%20the%20notifyCheckpointComplete%20callback.

Also, there is an issue opened for a long time to support iceberg connector without checkpointing: https://github.com/apache/iceberg/issues/1442

Oh wow, that’s unexpected. Thanks a lot for the information!

@Aleksandr Pilipenko When you say iceberg sink connector commits data to tables on checkpoint events, does it mean for any event a lag of up to the checkpoint interval is to be expected between flink and s3? If so, we would want to keep the checkpoint interval as low as possible (sub-second) to be near real-time. With low interval, we will be creating checkpoints that much more often even when there is no data flowing. Won't this create an issue while trying to compact all these checkpoints, not to mention a huge increase in s3 usage?

@Pankaj Joshi I think this question should be addressed to Iceberg community/maintainers. https://iceberg.apache.org/community/ When I say that Iceberg uses checkpoints as trigger to commit data to tables, I just pointing that connector implemented that way. Unfortunately I am not aware of reasons why this approach was chosen for this connector.

Thanks @Aleksandr Pilipenko for the response. I'll follow up /w iceberg community. At this point, I was just wondering if the behavior I mentioned is what is expected.