Am I going insane, or is a KafkaSource that restar...
# troubleshooting
b
Am I going insane, or is a KafkaSource that restarts from a savepoint with a
new
set of topics only adding to the topic/partitions it reads from ? Looking at this file https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/sr[…]nk/connector/kafka/source/enumerator/KafkaSourceEnumerator.java. My assumption is that when restarting from savepoint, the
assignedPartitions
are the partitions from the savepoint. It eventually gets to
chechPartitionChanges
which correctly identifies the new and removed partitions. If then eventually gets to [handlePartitionSplitChanges](https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/sr[…]nk/connector/kafka/source/enumerator/KafkaSourceEnumerator.java) where the new partitions are added, but there is a
Copy code
// TODO: handle removed partitions
even in the latest version of the enumerator. I noticed a weird behavior when I changed the topic list for one of my KafkaSources and did a stateful upgrade (take savepoint, restart from savepoint) and noticed in the consumer group that offsets for both topics (the old one and the new one) were committed on checkpoints/savepoints. I than took another savepoint and using the state analyzer api looked at the operator state from the savepoint and both topics are now listed in the savepoint. I may be missing something, but after noticing the behavior and looking at the source code I'm quite baffled. Using Flink 1.16.3 but based on the source code this may be also an issue in the latest version of the Kafka connector...