I recently notice a behaviour in our Kafka ingesti...
# general
j
I recently notice a behaviour in our Kafka ingestion tasks where when Kafka retention kicks in and the data Druid is trying to read is not available, Druid gets stuck in a loop like:
Copy code
2025-05-13T23:09:59,308 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.AbstractFetch - [Consumer clientId=consumer-kafka-supervisor-bmcfdhbl-1, groupId=kafka-supervisor-bmcfdhbl] Fetch position FetchPosition{offset=1189300635, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-broker:9092 (id: 1691 rack: null)], epoch=absent}} is out of range for partition datasource-124, raising error to the application since no reset policy is configured
2025-05-13T23:09:59,308 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=1189300635, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-broker:9092 (id: 1691 rack: null)], epoch=absent}} is out of range for partition datasource-124]
2025-05-13T23:09:59,309 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms
What I'm wondering is why isn't the Supervisor reporting the error? The job just gets stuck in an infinite loop...
The documentation here seems to indicate that the Supervisor will fail. However, the latest version of the documentation has no such mention of this.
j
I don't know about the looping behavior ... but have you tried resetOffsetAutomatically=true to instruct it to fix the partition offset and resume fetching?
j
The documentation states:
If set to
false
, Druid surfaces the exception causing tasks to fail and ingestion to halt.
^ this is the behaviour we want but it doesn't appear to be working
I should not we are on Druid 29.0.1
l
The docs from this config appear to have moved to the
supervisor
section. Probably because they are shared across different streaming types. https://druid.apache.org/docs/29.0.1/ingestion/supervisor 29.0.1 docs still state that tasks should fail.
to confirm the behavior you are seeing: Your kafka tasks log this in a loop once offsets are out of range. And then, when the task time is up, are the tasks passing or failing?
j
^ The tasks are passing