Hi, We have a flink job that is consuming from a k...
# troubleshooting
a
Hi, We have a flink job that is consuming from a kafka topic and writing to S3 in ORC format. We have recently changed the
auto.offset.reset
policy to earliest and started seeing that randomly some partition offset resetting to earliest offset with the following INFO log in the taskmanager
Copy code
Fetch offset 280306358700 is out of range for partition topic-0, resetting offset
Resetting offset for partition topic-0 to offset 280164596874.
Flink is connecting to kafka brokers using AWS NLB. I have also checked the topic directory and this offset is present. I also created a dummy consumer to consume from this partition and offset and I was able to consume the message Flink Version: 1.13.1 Platform: AWS EMR Kafka Version: 2.4.1 (Commit:c57222ae8cd7866b) Scala: 2.11 Java: 8