Hello, we would like to request some help from com...
# troubleshooting
n
Hello, we would like to request some help from committer at flink-connector-kafka. Due to project needs, we have made the following modification at: KafkaPartitionSplitReader.java as shown in the attached screen. So to pass some already processed records based on the nature of business logic we have use the kafkaConsumer.seek(....) [screen attached] Unfortunately that does not work ok, as after that seek(..) method it makes the committedOffsets going up n down in a crazy fashion as discussed here from @Konstantinos Karavitis: https://apache-flink.slack.com/archives/C03G7LJTS2G/p1741176740251049 The behavior is like Flink does not like that. Does anybody know why is that happening? so that we can adapt further and make that modification work. For example does Checkpointing is doing checksum, and because we do seek(...) we break the checksum and we have that weird behavior? Many thanks in advance!
p
Since Flink does not rely on active consumers from Kafka API, I'd say using seek may not be the best shot there indeed. Would a skip record do the trick instead of using seek?
n
hello thanks for investing the time and asnwering, i think i not get you with rely on active consumers, maybe you can explain further? From code is clear that KafkaSource is using KafkaConsumer<byte[], byte[]> to fetch the records. I will try your proposal, instead of seek, if that record.offset is inside that offset range [a-b] that has been processed before then just ignore-skip, unfortunately there will be some cpu cycle waste, but i think to the point we are this is unavoidable. but still strange how that seek is producing that behaviour
p
Flink’s KafkaSource does not rely on active consumers in the sense of Kafka’s traditional consumer groups. Instead of dynamically balancing partitions among active consumers, Flink manages partition assignments itself via its checkpoints and stateful recovery. This ensures exactly-once processing and fault tolerance, independent of Kafka’s consumer group rebalancing. So if you check if a consumer id from a Flink pipeline it will always be "inactive" on kafka. Flink’s Kafka consumers track offsets internally and restore from checkpoints rather than relying on Kafka’s automatic offset tracking.
n
thank you!