facing the weird issue in pinot (debugged more and found serialization exception in pinot logs and we see this message offset 39777 in partition 2 with wrong schema id , after getting this exception pinot stopped consuming further from partition 2 ), pinot stopped consuming from 2nd number partition and in propertystore I can see so many segments created and stuck on this offset :

{
  "id": "tSHistoricCalibration__2__220__20220715T1700Z",
  "simpleFields": {
    "segment.creation.time": "1657904457323",
    "segment.flush.threshold.size": "10000",
    "segment.realtime.numReplicas": "1",
    "segment.realtime.startOffset": "39777",
    "segment.realtime.status": "IN_PROGRESS"
  },
  "mapFields": {},
  "listFields": {}
}

We are using confluent kafka and schema registry and decoder is used KafkaConfluentSchemaRegistryAvroMessageDecoder. Do we have any configuration so that pinot can ignore such errors and consume further messages @Mayank @Chad

Hi @Chad, @Sandeep Dabade can you get us some help on this?

we had a similar issue with poison pills so i don't believe there is any config to skip - there is an open PR to fix the magic byte error (i.e. non avro message on an avro topic) - https://github.com/apache/pinot/issues/9045

That ticket will address skipping messages upon exceptions via table config in a separate PR. At the moment if the error persists the consumer will keep dying

Unfortunaltey, that’s the current behavior and fix is in works.