Hi #C03G7LJTS2G I have observed one issue/behaviour with Flink Iceberg streaming pipeline. I have checkpoint configured as 10 secs since the expected latency is 10 secs and I am able to get the records as expected but what I have seen is lot of Iceberg snapshots are also created for that table even if there is no data flowing in every 10 seconds. Checkpoints are working fine, no errors but I was expecting that if there is new data then only the snapshot should be written. So what am I missing here? Is this the expected behavior? Also, the other problem is that, I only inserted 3 records in my topic and the same can be seen in the target table but in the Flink UI I can see something as attached. Why am I seeing this?

To explain the screenshot: The actual data records/rows are written by

IcebergStreamWriter

in the first node already. The records that are being passed down to IcebergFilesCommitter in the second node (and therefore the ones that are counted in the UI) are metadata records about parquet files that the upstream writer has written, so you can expect records there at least every checkpoint even if there is no new data. As to why Iceberg snapshots/commits are created even though there's no new data, you can find an explanation here in this Github issue.

Got it thanks @Urs Schoenenberger