Hello everyone, I am using Flink to process data f...
# troubleshooting
đ
Hello everyone, I am using Flink to process data from Kafka and write it to Iceberg. I am facing two issues: First, the data is not being immediately added to Iceberg but only after a checkpoint is made, even though I am using AT_LEAST_ONCE mode. This causes significant delays when trying to build real-time applications. Secondly, I noticed that after 10 checkpoints, Flink commits additional empty metadata files to Iceberg. As a result, Flink jobs are generating a lot of metadata. Why is Flink committing so many empty files? Is there a way to fix this? Please help me resolve these issues.
r
What kind of real-time applications are you building? I don't think Iceberg is suited to that kind of use. You might want to look at tools like Clickhouse, Pinot, Druid, etc if you need that kind of low-latency.
Flink commits additional empty metadata files to Iceberg
might be worth looking at
write.metadata.previous-versions-max
and
write.metadata.delete-after-commit.enabled
settings. Also the flink channel over on the Apache Iceberg slack group for additional insight on the metadata files and impact of retaining them or not.
a
First, the data is not being immediately added to Iceberg but only after a checkpoint is made, even though I am using AT_LEAST_ONCE mode. This causes significant delays when trying to build real-time applications.
Iceberg commits data only after successful checkpoint, so this behaviour is by the Iceberg connector design. This however is not stated very clearly in the docs:
Iceberg commit happened after successful Flink checkpoint in the
notifyCheckpointComplete
callback.