I have a flink pipeline that reads from kafka, pro...
# troubleshooting
j
I have a flink pipeline that reads from kafka, processes this data in some Async flink function, and then produces it back to kafka. We are using the flink operator with the
upgradeMode: savepoint
- and we would expect on a job upgrade, the job to stop processing, create a full savepoint, and restart from said savepoint, thus processing every event only once. however, we are seeing duplicates in our pipeline output - anyone that can make sense of this?
Note: Our kafka sink is set to DeliveryGuarantee
AT_LEAST_ONCE
- I just dont see how this should impact on clean graceful shutdowns and job upgrades?
s
There will be some data that gets reprocessed during this upgrade interval. Savepoint operation might take time (couple of minutes some times) . Can to try exactly_once setting in kafka?
j
Interesting. I would have thought it finished all its work on savepoint trigger, and didnt take new work in. 😒
s
Lets say If a savepoint is triggered at time t and it completed at t+30s. Job doesnt stop processing after time t. So those 30s data needs to be reprocessed. If your sink doesnt have the capability to handle duplicates (using database like transactions ), u will end up having duplicates in ur sink
j
Thanks a lot πŸ™‚ I guess I just misunderstood how the operator shut down the job. Have a nice evening πŸ™‚
πŸ‘ 1