Hi Team Data not appearing in Pinot Query ...
# troubleshooting
r
Hi Team Data not appearing in Pinot Query Console. I am pushing data to Pinot through kafka, By command bin/kafka-console-producer.sh --broker-list localhost:19092 --topic mytopic < $PDATA_HOME/data.json I check all logs, there is no exceptions, but my data not appearing in query tool. Need Help 🙂 My Schema look like this:
Copy code
{
  "schemaName": "eventflowstats",
  "dimensionFieldSpecs": [
    {
      "name": "_index",
      "dataType": "STRING"
    },
    {
      "name": "_type",
      "dataType": "STRING",
      "maxLength": 5
    },
    {
      "name": "_id",
      "dataType": "STRING"
    },
     {
      "name": "_source.aExpIds",
      "dataType": "INT",
      "singleValueField": false
    }
	]
  "dateTimeFieldSpecs": [
    {
      "name": "_source.sDate",
      "dataType": "LONG",
      "format": "1:SECONDS:SIMPLE_DATE_FORMAT:SECONDS:SIMPLE_DATE_FORMAT",
      "granularity": "1:DAYS"
    }
	]
}
My Table Config like this:
Copy code
{
  "tableName": "mytable",
  "tableType": "REALTIME",
  "tenants": {},
  "segmentsConfig": {
    "timeColumnName": "_source.sDate",
    "timeType": "MILLISECONDS",
    "segmentPushType": "APPEND",
    "replicasPerPartition": "1",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "1"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowLevel",
      "stream.kafka.topic.name": "mytopic",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.hlc.zk.connect.string": "localhost:2191/kafka",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.zk.broker.url": "localhost:2191/kafka",
      "stream.kafka.broker.list": "localhost:19092"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
And Data like this:
Copy code
{
  "_index": "dhfkdfkdsjfk",
  "_type": "_doc",
  "_id": "68767677989hjhjkhkjh",
  "_source.aExpIds": [
    815850,
    815857,
    821331
  ],
  "_source.sDate": "2021-01-04 00:00:00"
}
I check all logs, I did not find any exceptions. But data is not appearing in Pinot controller portal.
x
Kafka topic
event-count-stats-topic
you are producing to and the topic in your table configs are not same
"stream.kafka.topic.name": "mytopic",
r
sorry, both I am using same, here(post) I forgot to change both places as mytopic.
x
the schema format is wrong here
Copy code
dateTimeFieldSpecs": [
    {
      "name": "_source.sDate",
      "dataType": "LONG",
      "format": "1:SECONDS:SIMPLE_DATE_FORMAT:SECONDS:SIMPLE_DATE_FORMAT",
      "granularity": "1:DAYS"
    }
r
can correct me please
x
Copy code
"format": "1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-dd HH:mm:ss",
r
i thinks , "dataType: LONG" also wrong? it should be STRING , am I right?
x
right