Hey all,
For some reason when using the KafkaSink...
# troubleshootingo
Or Keren
07/05/2023, 9:21 AMHey all,
For some reason when using the KafkaSink with DELIVERY_ONCE and restarting the application without a checkpoint (it had a
last-statestatelessm
Martijn Visser
07/05/2023, 9:23 AMDid you set a new
setTransactionalIdPrefixo
Or Keren
07/05/2023, 9:24 AMNo, it's the same TransactionalIdPrefix as before.
Plus when changing it, it's quickly initialized.
m
Martijn Visser
07/05/2023, 9:25 AMThe situation that you're now in, is that you have lingering transactions that aren't correctly aborted. That's also what you see from the logs
o
Or Keren
07/05/2023, 9:26 AMWhat is the best approach here?
m
Martijn Visser
07/05/2023, 9:27 AMIf you really want to start fresh, then you should basically start as a completely new job.
Martijn Visser
07/05/2023, 9:28 AMIs your source Kafka too?
o
Or Keren
07/05/2023, 9:28 AMYes
Or Keren
07/05/2023, 9:28 AMI'm trying to understand why it's taking so long, the once's that wasn't aborted are probably the ones from the last checkpoint that was not completed, no? It should be less than the parallelism count
m
Martijn Visser
07/05/2023, 9:28 AMIt's not only the last checkpoint, but can be for multiple
o
Or Keren
07/05/2023, 9:29 AMI configured only one checkpoint in parallel, so the rest of them completed successfully
m
Martijn Visser
07/05/2023, 9:30 AMMost likely you should be good, in theory there might be a lingering transaction but that's most likely not the case for you
Martijn Visser
07/05/2023, 9:31 AMBut transaction aborting is always expensive and takes a long time
o
Or Keren
07/05/2023, 9:31 AMIf I will start a completely new job, will it fix that?
m
Martijn Visser
07/05/2023, 9:32 AMIt should yes, you're basically a new consumer and producer in that case
Martijn Visser
07/05/2023, 9:32 AMAs long as you make sure that all references to things like consumer groups, transaction ID prefixes etc are all different
o
Or Keren
07/05/2023, 9:33 AMHmm that's a problem actually
Or Keren
07/09/2023, 2:15 PMIn the end that's what we did, change the prefix whenever we start the application without a checkpoint 🙂
Or Keren
07/09/2023, 2:17 PMMaybe it will be good to suggest this as a feature that's configurable, whether we want to change the prefix on starting the application in stateless mode, that way it won't need to abort lingering transactions and will save a lot of time
m
Martijn Visser
07/10/2023, 10:08 AMThat’s a conflicting result: you want exactly once but then you want to restart in stateless mode. Those things are conflicting with each other
o
Or Keren
07/10/2023, 2:51 PMI don't agree, since you can start with a clean state but continue from the last committed message in Kafka, you will read and process only once each message even if you switch prefix.
2 Views