I am facing the same issue as described in this st...
# troubleshootingz
Zhong Chen
05/24/2023, 10:56 PMI am facing the same issue as described in this stackflow post. Anyone knows what might cause this? Flink 1.17, Java 11, Scala 2.12.17 https://stackoverflow.com/questions/75940903/flink-kafkasink-takes-long-time-to-start
m
Martijn Visser
05/25/2023, 4:57 AMFlink doesn’t support Scala 2.12.17
Martijn Visser
05/25/2023, 4:58 AMIs this also a new job that you are starting, or restarting from a savepoint?
z
Zhong Chen
05/25/2023, 5:55 AMFrom a savepoint. I think it is due to some corrupted states of kafka transactions. After I cleared the state by removing
eosm
Martijn Visser
05/25/2023, 8:03 AMHow old was the savepoint that you tried to recover from?
z
Zhong Chen
05/25/2023, 4:02 PMIt was pretty recent. It started to happen after an upgrade. I use
flink-k8s-operatorZhong Chen
05/25/2023, 5:16 PMso it happens again after an upgrade.
t
Tzu-Li (Gordon) Tai
05/25/2023, 9:05 PM@Zhong Chen I believe the long startup time is caused by a loop at restore time that attempts to abort lingering transactions from previous executions of the job. Flink’s KafkaSink currently doesn’t have a good way to directly query Kafka and list lingering transactions, so instead it needs to iterate through all possible permutations of transaction IDs to abort and try to abort them. We’re collaborating with folks on the Kafka side to see if we can improve this.
Tzu-Li (Gordon) Tai
05/25/2023, 9:05 PMjust curious: did you start seeing the long startup times only after upgrading to 1.17? Or was this observed in older versions as well?
z
Zhong Chen
05/25/2023, 9:18 PMafter upgrading to 1.17 I believe
Zhong Chen
05/25/2023, 9:48 PMI am just starting using Flink, so I can’t confidently say it only happens to 1.17. I only briefly used 1.16. Since I noticed that auto scaling is only applicable to 1.17, I decided to use the latest version.