Ayush Chauhan
05/22/2024, 10:04 AMauto.offset.reset policy to earliest and started seeing that randomly some partition offset resetting to earliest offset with the following INFO log in the taskmanager
Fetch offset 280306358700 is out of range for partition topic-0, resetting offset
Resetting offset for partition topic-0 to offset 280164596874.
Flink is connecting to kafka brokers using AWS NLB. I have also checked the topic directory and this offset is present. I also created a dummy consumer to consume from this partition and offset and I was able to consume the message
Flink Version: 1.13.1
Platform: AWS EMR
Kafka Version: 2.4.1 (Commit:c57222ae8cd7866b)
Scala: 2.11
Java: 8