Đức Hân Trần
10/01/2024, 1:34 AMrmoff
10/01/2024, 9:07 AMFlink commits additional empty metadata files to Icebergmight be worth looking at
write.metadata.previous-versions-max and write.metadata.delete-after-commit.enabled settings. Also the flink channel over on the Apache Iceberg slack group for additional insight on the metadata files and impact of retaining them or not.Aleksandr Pilipenko
10/01/2024, 9:56 AMFirst, the data is not being immediately added to Iceberg but only after a checkpoint is made, even though I am using AT_LEAST_ONCE mode. This causes significant delays when trying to build real-time applications.Iceberg commits data only after successful checkpoint, so this behaviour is by the Iceberg connector design. This however is not stated very clearly in the docs:
Iceberg commit happened after successful Flink checkpoint in thecallback.notifyCheckpointComplete