Hi Guys I stood up pinot cluster with in one node and create Apache Pinot #troubleshooting

Hi Guys, I stood up pinot cluster with in one node...

Sandeep R

10/31/2021, 11:30 PM

Hi Guys, I stood up pinot cluster with in one node, and created schema and integrated kafka broker to it. I tried inserting some sample data, but it is falling to consumer from pinot, getting below error,

Copy code

), currentOffset=1939499, numRowsConsumedSoFar=1, numRowsIndexedSoFar=1
2021/10/31 23:08:26.026 ERROR [LLRealtimeSegmentDataManager_pnrevents__0__0__20211031T2306Z] [pnrevents__0__0__20211031T2306Z] Caught exception while transforming the record: {
  "fieldToValueMap" : {
    "pt" : null,
    "osia" : null,
    "excludePII" : 0,
    "pcc" : null,
    "lname" : null,
    "mel" : null,
    "tkt" : null,
    "kafkaProps" : null,
    "timestamp" : null,
    "ver" : null,
    "dts" : null,
    "proxyUrl" : "<http://xyz.com:8080|xyz.com:8080>",
    "docid" : null,
    "rlc" : null,
    "rcode" : null,
    "message" : {
      "jsver" : "1",
      "core" : "1g",
      "pcc" : "ABCD",
      "notif" : "abc",
      "ver" : "5",
      "dts" : "20211022",
      "lname" : [ "JONES" ],
      "pt" : "2021-10-22T02:22:48.196",
      "mel" : [ "MULTIPAX.V6@XYZ.COM" ],
      "docid" : "1a0343cdrdc455",
      "rlc" : "1234F"
    },
    "itina" : null,
    "gname" : null,
    "url" : "<https://xyz.com/pir/abc>",
    "agencyName" : "MTT",
    "securityToken" : "xxxxxxxxxxxxx",
    "jsver" : null,
    "core" : null,
    "notif" : null,
    "nama" : null,
    "npnr" : null,
    "kafka" : null,
    "emd" : null,
    "phonea" : null
  },
  "nullValueFields" : [ ]
}
java.lang.RuntimeException: Caught exception while transforming data type for column: message
        at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:120) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.segment.local.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:82) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:510) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:417) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:560) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalStateException: Cannot read single-value from Collection: [1, 1g, ABCD, abc, 5, 20211022, [Ljava.lang.Object;@4049e608, 2021-10-22T02:22:48.196, [Ljava.lang.Object;@3eb034ea, 1, 1234F] for column: message
        at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:721) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.standardizeCollection(DataTypeTransformer.java:199) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.standardize(DataTypeTransformer.java:144) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:90) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]

Ken Krugler

11/01/2021, 2:06 AM

Hi Sandeep - what’s the schema?

Sandeep R

11/01/2021, 2:27 AM

Copy code

{
  "schemaName": "pnrevents",
  "dimensionFieldSpecs": [
    {
      "name": "url",
      "dataType": "STRING"
    },
    {
      "name": "proxyUrl",
      "dataType": "STRING"
    },
    {
      "name": "securityToken",
      "dataType": "STRING"
    },
    {
      "name": "excludePII",
      "dataType": "BOOLEAN"
    },
    {
      "name": "kafka",
      "dataType": "STRING"
    },
    {
      "name": "kafkaProps",
      "dataType": "STRING"
    },
    {
      "name": "agencyName",
      "dataType": "STRING"
    },
    {
      "name": "message",
      "dataType": "STRING"
    },
    {
      "name": "notif",
      "dataType": "STRING"
    },
    {
      "name": "jsver",
      "dataType": "STRING"
    },
    {
      "name": "docid",
      "dataType": "STRING"
    },
    {
      "name": "rlc",
      "dataType": "STRING"
    },
    {
      "name": "core",
      "dataType": "STRING"
    },
    {
      "name": "ver",
      "dataType": "STRING"
    },
    {
      "name": "dts",
      "dataType": "STRING"
    },
    {
      "name": "pt",
      "dataType": "STRING"
    },
    {
      "name": "pcc",
      "dataType": "STRING"
    },
    {
      "name": "rcode",
      "dataType": "STRING"
    },
    {
      "name": "osia",
      "dataType": "STRING"
    },
    {
      "name": "itina",
      "dataType": "STRING"
    },
    {
      "name": "nama",
      "dataType": "STRING"
    },
    {
      "name": "npnr",
      "dataType": "STRING"
    },
    {
      "name": "phonea",
      "dataType": "STRING"
    },
    {
      "name": "lname",
      "dataType": "STRING"
    },
    {
      "name": "gname",
      "dataType": "STRING"
    },
    {
      "name": "mel",
      "dataType": "STRING"
    },
    {
      "name": "tkt",
      "dataType": "STRING"
    },
    {
      "name": "emd",
      "dataType": "STRING"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "timestamp",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}

Ken Krugler

11/01/2021, 2:30 AM

I’m assuming the Kafka scheme has

message

as something other than a simple string, and that’s why the conversion is failing.

Sandeep R

11/01/2021, 2:31 AM

my message structure is a JSON

Sandeep R

11/01/2021, 2:31 AM

let me send you JSON sample

Ken Krugler

11/01/2021, 2:32 AM

So the

message

field will be parsed as a map, and you’re trying to save it into a string, yes?

Sandeep R

11/01/2021, 2:33 AM

Copy code

{
  "url": null,
  "proxyUrl": "<http://xyz.com:8080|xyz.com:8080>",
  "securityToken": "ZZZZZZZZZZZZZZZZZZZZZZZZZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "excludePII": false,
  "kafka": "tripxyz",
  "kafkaProps": null,
  "agencyName": "tripxyz",
  "message": {
    "notif": "pnr",
    "jsver": "1",
    "docid": "1gabcdef20211022",
    "rlc": "ABCDEF",
    "core": "1g",
    "ver": "0",
    "dts": "20211022",
    "pt": "2021-10-22T02:21:31.760",
    "pcc": "ABCD",
    "rcode": {
      "osia": "Y",
      "itina": "Y",
      "nama": "Y",
      "npnr": "Y",
      "phonea": "Y"
    },
    "lname": [
      "JONES"
    ],
    "gname": null,
    "mel": [
      "MULTIPAX.V6@XYZ.COM"
    ],
    "tkt": [],
    "emd": []
  }
}

Ken Krugler

11/01/2021, 2:33 AM

Take a look at https://docs.pinot.apache.org/basics/data-import/complex-type

Sandeep R

11/01/2021, 2:37 AM

yes, message is mapping, it's a nested json.

Ken Krugler

11/01/2021, 2:55 AM

Then (as per the link I sent) you’ll need to set up your Pinot table schema to either flatten it, or use a JSON index for that field.

Ken Krugler

11/01/2021, 2:56 AM

Though there’s probably a way to map it to a simple string field, I just don’t know how, sorry.