This message was deleted.

Can you share a task log with the exception. What Druid version are you on?

We are on v25.0.0 logs here

2023-04-05T00:55:23,310 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_private.DATAPLATFORMV2.events_3f76be529bcb647_gcgcihkh",
  "status" : "FAILED",
  "duration" : 3397,
  "errorMsg" : "org.apache.druid.java.util.common.parsers.ParseException: Failed to parse avro message.\n\tat org.apac...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
2023-04-05T00:55:23,315 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2023-04-05T00:55:23,327 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2023-04-05T00:55:23,331 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2e60d4f4{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2023-04-05T00:55:23,332 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2023-04-05T00:55:23,333 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@272485a6{/,null,STOPPED}
2023-04-05T00:55:23,336 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2023-04-05T00:55:23,336 INFO [main] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='10.40.89.93:8100', hostAndPort='10.40.89.93:8100', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
2023-04-05T00:55:23,336 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
2023-04-05T00:55:23,337 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
2023-04-05T00:55:23,337 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_private.DATAPLATFORMV2.events_3f76be529bcb647_gcgcihkh].
2023-04-05T00:55:23,337 INFO [main] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [READING])
2023-04-05T00:55:23,406 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped.
2023-04-05T00:55:23,408 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2023-04-05T00:55:23,513 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10005abdb2e053b
2023-04-05T00:55:23,513 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x10005abdb2e053b closed
2023-04-05T00:55:23,517 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task

I'm looking into it. I have the same expectation you do. It should skip the parse errors, I need to spend some time in the code to try to understand where it might be going wrong. Can you share your ingestion spec?

here is the spec., pretty much standard except that we use a custom avro decoder that we wrote to handle avro schema embedded in the message rather than inline or registry based.

{
  "type": "kafka",
  "spec": {
    "ioConfig": {
      "type": "kafka",
      "consumerProperties": {
        ....
      },
      "topic": "topic-abc",
      "inputFormat": {
        "type": "kafka",
        "valueFormat": {
          "type": "avro_stream",
          "avroBytesDecoder": {
            "type": "glo_schema" #this is our custom decoder, where we process schema that is embedded in the message, and throws ParseException based on schema/payload validation
          }
        },
        "flattenSpec": {
          "useFieldDiscovery": true
        },
        "binaryAsString": false
      },
      "appendToExisting": false,
      "useEarliestOffset": true,
      "idleConfig": {
        "enabled": true,
        "inactiveAfterMillis": 300000
      }
    },
    "tuningConfig": {
      "type": "kafka",
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0
    },
    "dataSchema": {
      .....
    }
  }
}

Here's what I found. I'm not sure this is the reason, I think a committer may need to take a look: https://github.com/apache/druid/blob/58a3acc2c4c5d5d22098e15f7ff25d6e9baa8541/inde[…]org/apache/druid/indexing/seekablestream/StreamChunkParser.java I'm wondering whether the byteEntityReader.read() is referencing you reader and when/if it throws the ParseException here it would not use the ParseExceptionHandler. Not sure what the solution is, but I think this may be the cause.

@Sergio Ferragut as you mentioned, apparently looks like its not hitting the handler.

2023-04-06T04:38:07,887 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
org.apache.druid.java.util.common.parsers.ParseException: Failed to parse avro message from this origin:unknown
	at org.apache.druid.data.input.avro.GlobalityAvroBytesDecoder.parse(GlobalityAvroBytesDecoder.java:111) ~[?:?]
	at org.apache.druid.data.input.avro.AvroStreamReader.intermediateRowIterator(AvroStreamReader.java:70) ~[?:?]
	at org.apache.druid.data.input.IntermediateRowParsingReader.intermediateRowIteratorWithMetadata(IntermediateRowParsingReader.java:231) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.IntermediateRowParsingReader.read(IntermediateRowParsingReader.java:49) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.kafkainput.KafkaInputReader.buildBlendedRows(KafkaInputReader.java:158) ~[?:?]
	at org.apache.druid.data.input.kafkainput.KafkaInputReader.read(KafkaInputReader.java:129) ~[?:?]
	at org.apache.druid.segment.transform.TransformingInputEntityReader.read(TransformingInputEntityReader.java:43) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SettableByteEntityReader.read(SettableByteEntityReader.java:70) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.StreamChunkParser.parseWithInputFormat(StreamChunkParser.java:135) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.StreamChunkParser.parse(StreamChunkParser.java:104) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:635) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:266) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:151) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 36 out of bounds for length 2
	at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:460) ~[?:?]
	at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:284) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) ~[?:?]
	at org.apache.druid.data.input.avro.GlobalityAvroBytesDecoder.parse(GlobalityAvroBytesDecoder.java:95) ~[?:?]
	... 19 more

Right. So it seems like whenever a byteEntityDecoder is used it will have this behavior. I've created a bug report on this here: https://github.com/apache/druid/issues/14041

Thank you @Sergio Ferragut