Hi team, I have a question about the Flink CDC → Iceberg sink. Is there a way to periodically roll or flush Iceberg data files based on file size or record count, independent of the checkpoint interval? I want to increase the checkpoint interval to 2–3 minutes, but I don’t want all the data to remain buffered in the TaskManager until the checkpoint completes. Ideally, I’d like to have multiple files written within a single checkpoint based on size or record count. I tried the following properties, but they didn’t seem to have any effect:

sink:
  type: iceberg
  table.properties.write.target-file-size-bytes: "4194304"
  table.properties.write.parquet.row-group-size-bytes: "4194304"
  table.properties.write.parquet.row-group-check-max-record-count: "10000"

A large batch arriving within one checkpoint still gets written as a single file, with the file size depending on how much data arrived during that checkpoint. Does the Flink CDC pipeline Iceberg connector have any parameter to periodically flush or roll writer files by size or record count, independent of the checkpoint interval? Or is the checkpoint interval the only way to control the write frequency?

Thanks for reporting this issue. We confirmed that the Flink CDC Iceberg sink currently uses a hard-coded target file size and does not fully propagate the configured Iceberg write properties to the runtime writer. Therefore, these properties do not affect file rolling as expected. We agree that files should be able to roll by size within a single checkpoint, while still being committed atomically when the checkpoint succeeds. We have created FLINK-40651 to track this improvement and plan to address it. Thanks for the feedback!

@kunni Thanks for looking into this and creating FLINK-40651. I’d be happy to work on this improvement. Could you please assign the issue to me?

Yeah, assigned.

Will try to close this by this weekend