Hi Flinkers, When using the Kafkasink with `<http:...
# troubleshooting
j
Hi Flinkers, When using the Kafkasink with
DeliveryGuarantee.AT_LEAST_ONCE
and the job performs a clean restart - without any error or warnings - would it then be expected to produce duplicates. I see duplicates, but I would expect the KafkaSink to "drain" finish remaining/in-flight work and checkpoint/safepoint the rest. Is my assumption wrong that there "should" not be duplicates with
DeliveryGuarantee.AT_LEAST_ONCE
on a "clean" restart?
p
would it then be expected to produce duplicates
Likely yes. Flink only commits to kafka offsets after a checkpoint. So if you’re not running a savepoint on the shutdown phase, you’ll get them.
Is my assumption wrong that there “should” not be duplicates with
DeliveryGuarantee.AT_LEAST_ONCE
on a “clean” restart?
The at least once should always lead you to a possible duplicates. Committing offsets may fail as well.