Jonas Brunsgaard
06/18/2024, 11:01 AMDeliveryGuarantee.AT_LEAST_ONCE and the job performs a clean restart - without any error or warnings - would it then be expected to produce duplicates. I see duplicates, but I would expect the KafkaSink to "drain" finish remaining/in-flight work and checkpoint/safepoint the rest.
Is my assumption wrong that there "should" not be duplicates with DeliveryGuarantee.AT_LEAST_ONCE on a "clean" restart?Pedro Mázala
06/18/2024, 2:03 PMwould it then be expected to produce duplicatesLikely yes. Flink only commits to kafka offsets after a checkpoint. So if you’re not running a savepoint on the shutdown phase, you’ll get them.
Is my assumption wrong that there “should” not be duplicates withThe at least once should always lead you to a possible duplicates. Committing offsets may fail as well.on a “clean” restart?DeliveryGuarantee.AT_LEAST_ONCE