does anyone know if there is a way to handle ordering issues w.r.t debezium incremental snapshots (via kafka and not the flink cdc connector) in flink? In case of incremental snapshots there can be cases when the read events might come sooner than an update event that occurred before the read.

Is it possible that your data is not properly partitioned. E.g in the Kafka topic between debezium and Flink, there's no partitioning key defined, so data is distributed randomly across all Kafka partitions? If you have a primary key in your source table, use the same key for your Kafka topic, then events will be stored in order in Kafka partitions, per primary key.

thanks for the reply @rmetzger it’s not about the partitions. The problem is with running incremental snapshots. Debezium incremental snapshots are where along with regular CDC it keeps reading rows for older data. Now it is possible that the CDC job is lagging as it is catching up with updates, while the incremental snapshot read a row in it’s current state while it’s update events are still to be processed by debezium. Let’s assume Time=1 and Time=2 is where updates happended and an incremental snapshot read the same row at time=3. In the kafka topic the events will come in this order -> READ(time=3) -> Update(time=1) -> Update(time=2). This would cause issues when doing stateful computations.

Hey, so in Debezium's incremental snapshotting implementation, snapshot events and log-based events are interleaved, applying some means of windowed de-duplication. What could happen is one of these three cases: 1: you get an update and then a read, reflecting the same state, for the same record later on (when both occurred in different chunk windows) 2: you only get the update, but not the read (if they both occurred in the same chunk window and then the read event will be removed by the deduplicator) 3: you get the read and then an update, if they occurred in different chunk windows. I can't think of a case where an event would "overtake" another one, as you seem to suggest. If it happens, I'd consider that a bug.

thanks for explaining this, I think I missed reading the de-deuplication part in Debezium blog, reading it clarified my doubts: https://debezium.io/blog/2021/10/07/incremental-snapshots/

Perfect. I also did a talk about incremental snapshots in Debezium at Current last year. It discusses the feature and how it works some more: .

thanks, will watch it