Thanks <@UDQU92KBK>, I tried AvroSchemaToPinotSche...
# general
s
Thanks @User, I tried AvroSchemaToPinotSchema utility but it is giving error,
Copy code
Caused by: java.lang.IllegalStateException: Not one field in the RECORD schema at shaded.com.google.common.base.Preconditions.checkState
this error doesn't make sense as schema is correct and working with our messages.
👀 1
m
Are all the columns you’d like to read into Pinot nested?
s
I just need one column where I want to dump whole message in json format. I am able to add different column for every nested object from message but I dont want to do that because our schema has many nested fields which has further fields in it. We want whole message as it is in one column, is it possible to do with transformationConfig. For now what I did like below:
Copy code
"ingestionConfig": {
      "transformConfigs": [
        {
          "columnName": "header_json",
          "transformFunction": "jsonFormat(header)"
        },
        {
          "columnName": "dact_json",
          "transformFunction": "jsonFormat(dact)"
        },
        {
          "columnName": "ref_json",
          "transformFunction": "jsonFormat(refData)"
        }
        }
]
Avro schema is like :
Copy code
{
  "fields": [
  {
      "name": "header",
      "type": {
        "fields": [
          {
            "name": "fid",
            "type": "string"
          },
          {
            "name": "timestamp",
            "type": "long"
          }
        ],
        "name": "ABCHeader",
        "type": "record"
      }
    },
    {
      "name": "status",
      "type": {
        "name": "NNStatus",
        "namespace": "com.common",
        "symbols": [
          "N200_SUCCESS",
          "N400_SERVER_ERR"
        ],
        "type": "enum"
      }
    },
	{
      "name": "dact",
      "type": {
        "fields": [
          {
            "name": "lid",
            "type": "string"
          },
          {
            "name": "timestamp",
            "type": "long"
          }
        ],
        "name": "DactHeader",
        "type": "record"
      }
    },
    {
      "name": "defId",
      "type": "string"
    },
    {
      "name": "fixTimestamp",
      "type": "long"
    },
    "name": "CaRecord",
  "namespace": "com.mbn",
  "type": "record"
}
m
For whole message, as Kishore mentioned, you could use JSON index. But it would be good to understand how you want to query that JSON.