Hey guys, this is my first time implementing Pinot...
# troubleshooting
r
Hey guys, this is my first time implementing Pinot. I'm pointing to an existing Kafka and getting a decoding error.
Copy code
ERROR [JSONMessageDecoder] Caught exception while decoding row, discarding row. Payload is ��{"id":44,"is_active":true,"created_at":1734992620933,"__deleted":"false"}
org.apache.pinot.shaded.com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 2]
This is my schema:
Copy code
{
  "schemaName": "channel_schema",
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    },
    {
      "name": "is_active",
      "dataType": "BOOLEAN"
    },
    {
      "name": "__deleted",
      "dataType": "STRING"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "created_at",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}
This is my config:
Copy code
{
  "tableName": "channel_table",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "created_at",
    "schemaName": "channel_schema",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
		 "nullHandlingEnabled": true,
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "topic.name",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
			
      "stream.kafka.broker.list": "<kafka-host>",
			
			"security.protocol": "SASL_SSL",
      "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"<kafka-key>\" password=\"<kafka-secret>";",
      "sasl.mechanism": "PLAIN",
			
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
This topic is already used by other applications and converted to json without a problem.
m
There does seem to be some special character in your payload. Pinot uses fasterxml.jackson, do you know what JSON library those other systems use?
Copy code
��{"id":44,"is_active":true,"created_at":1734992620933,"__deleted":"false"}
r
I don't know what JSON library they use, but for a Databricks job they have, they are using io.confluent.kafka lib (I don't know if it helps with something) I also tried to pass the schema-registry on the config, but it didn't work. It's not a special character only in this message because I'm getting this to all topics from this cluster.