Davide Reich
03/05/2025, 1:34 PMimport org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
this.offsetsInitializer = OffsetsInitializer.earliest();
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
public static <T> KafkaSource<T> createKafkaSource(String bootstrapServers, String[] topics, String groupId, String clientIdPrefix, OffsetsInitializer offsetsInitializer, DeserializationSchema<T> deserializationSchema) {
return KafkaSource.<T>builder()
.setBootstrapServers(bootstrapServers)
.setTopics(topics)
.setGroupId(groupId)
.setClientIdPrefix(clientIdPrefix)
.setStartingOffsets(offsetsInitializer)
.setValueOnlyDeserializer(deserializationSchema)
.setProperties(getKafkaProperties())
.build();
}
The offset is set to earliest(), but when I try to reset the offset from Kafka UI, it doesn’t trigger the Flink job to reprocess the messages.
Has anyone encountered this issue or have any suggestions on how to properly manage the offset reset for a Flink job consuming from Kafka?Aleksandr Pilipenko
03/05/2025, 1:47 PMNote that Kafka source does NOT rely on committed offsets for fault tolerance. Committing offset is only for exposing the progress of consumer and consuming group for monitoring.https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/kafka/#consumer-offset-committing
Aleksandr Pilipenko
03/05/2025, 1:48 PMPedro Mázala
03/06/2025, 10:53 PMDavide Reich
03/09/2025, 6:11 PMPedro Mázala
03/10/2025, 2:54 PMDavide Reich
03/10/2025, 11:48 PM