I'm facing an issue with the Kafka source in my Fl...
# troubleshooting
d
I'm facing an issue with the Kafka source in my Flink job. In Kafka UI, I can see the Flink consumer and its offsets. However, when I try to manually reset the offset (moving it backward), it seems to get reset to zero, and the Flink job doesn't reprocess the messages. I’m using the following code to set up the Kafka source:
Copy code
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
this.offsetsInitializer = OffsetsInitializer.earliest();

import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;

public static <T> KafkaSource<T> createKafkaSource(String bootstrapServers, String[] topics, String groupId, String clientIdPrefix, OffsetsInitializer offsetsInitializer, DeserializationSchema<T> deserializationSchema) {
    return KafkaSource.<T>builder()
            .setBootstrapServers(bootstrapServers)
            .setTopics(topics)
            .setGroupId(groupId)
            .setClientIdPrefix(clientIdPrefix)
            .setStartingOffsets(offsetsInitializer)
            .setValueOnlyDeserializer(deserializationSchema)
            .setProperties(getKafkaProperties())
            .build();
}
The offset is set to
earliest()
, but when I try to reset the offset from Kafka UI, it doesn’t trigger the Flink job to reprocess the messages. Has anyone encountered this issue or have any suggestions on how to properly manage the offset reset for a Flink job consuming from Kafka?
a
Hi, Kafka source does not use offsets committed on Kafka side, rather utilizes offsets stored in source state.
Note that Kafka source does NOT rely on committed offsets for fault tolerance. Committing offset is only for exposing the progress of consumer and consuming group for monitoring.
https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/kafka/#consumer-offset-committing
If you want to start from offsets stored in consumer group - you will need to restart your Flink application without state
p
Hey there! What Aleksandr said is it indeed. Another way could be having a different consumer group to do so if you need to reeconsume with the previous state.
d
What is the best practice? In my application, there are multiple JARs that start different jobs depending on the context. As a result, jobs often need to be stopped and redeployed with updated JARs after an update. Most likely, messages will need to be reprocessed with the new update to apply any changes, right? So, does it make sense to reset the offset? I’m not sure what the best approach is.
p
So you want to have a way to: • upgrade the current code • restart from a checkpoint/savepoint consuming from offsets-X, where X is how much you want to reprocess • rely on saved/ha data Is this the gist?
d
That’s correct—I do want messages to be reprocessed, but I’d like to choose the offset myself by resetting the consumer group offset for the Flink job in Kafka. The issue is that when I redeploy the updated job, it likely picks up the last saved offset in Flink.