Hi everyone, I ran into a checkpoint recovery issu...
# flink-cdc
l
Hi everyone, I ran into a checkpoint recovery issue with the Fluss pipeline source: if a referenced partition has been dropped or expired, the source may fail to resume consumption even though the table and other partitions still exist. I opened FLINK-41313 to track the issue and submitted a fix in PR #4608. Local validation passed on Flink 1.20.3 / Java 11 and Flink 2.2.0 / Java 17, including checkpoint recovery ITs after partition deletion with both
earliest
and
full
startup modes. Could a maintainer please approve the CI workflow run and help review the PR? Thanks!
a
@litiliu this sounds like the similar issue which DDB and kinesis connectors had seen. Reference PR here: github.com/apache/flink-connector-aws/pull/193 I wonder if we have been able to priroritize a long term fix in the checkpoints itself? @Arvid Heise was driving that earlier
l
@Abhi Gupta Thanks for the reference. To clarify, our case does not involve a checkpoint consistency race. The checkpoint correctly contains splits for Fluss partitions that existed when it was taken. Those Fluss partitions can be dropped or expire afterwards. On restore, the Fluss connector attempts to subscribe to those now-missing partitions, preventing consumption from resuming. The fix reconciles restored splits with the current partition inventory before subscription. This external partition-removal scenario would still need connector-side handling even with consistent reader/enumerator checkpoints.
πŸ‘ 1