Hi team, We’re currently exploring Flink CDC for i...
# troubleshooting
v
Hi team, We’re currently exploring Flink CDC for ingesting data into Iceberg, and one of the areas we’re investigating is compaction. The compaction approach we currently use for Spark-based ingestion doesn’t seem to work efficiently here, mainly because of frequent commits and continuous ingestion, which result in a large number of small files. I wanted to understand what the standard approach is for handling compaction in this scenario. A couple of approaches we’re considering are: • Delayed compaction: Leave recent partitions untouched and compact only partitions that are at least X days old, as they’re less likely to receive new data. • Branch-based approach: Ingest data into one Iceberg branch and perform compaction and reads on another branch. Has anyone implemented either of these approaches, or found a more effective way to handle compaction with continuous Flink CDC ingestion? Please share your experiences or suggestions. Thanks!