Hey team I ve got an Avro schema which contains an array of Apache Pinot #troubleshooting

Hey team, I've got an Avro schema which contains a...

Dan DC

11/09/2021, 2:17 PM

Hey team, I've got an Avro schema which contains an array of records in a child field. I want to convert this to JSON during ingestion so I've added a transformation for this column to my realtime table. I've specified

as my complex type delimiter because I've got some Groovy transformations that I need to apply to other columns and is the only delimiter I can use to make my field names compatible with Groovy identifiers. My config looks like:

Copy code

...
"complexTypeConfig": {
  "delimiter": "$",
  ...
},
"transformConfigs": [
  ...
  "columnName": "some_field",
  "transformFunction": "json_format(parent_field$some_field)"
  ...
],
...

Dan DC

11/09/2021, 2:18 PM

This is not working for me. It seems messages are dropped because the json_format function can't be applied

Dan DC

11/09/2021, 2:18 PM

Any pointers on how I should do this?

Dan DC

11/09/2021, 2:20 PM

By the way, I've tried using

__

(double underscore) as my complex type delimiter but the complex type transformer didn't like that and was unable to extract the values resulting in null columns. I didn't spot anything strange in the code so I was wondering why this delimiter can't be used

Kishore G

11/09/2021, 2:41 PM

i think $ sign is causing some issues

Kishore G

11/09/2021, 2:41 PM

try escaping

Dan DC

11/09/2021, 6:35 PM

The endpoint rejects $ and wont let me create the table. I've tried adding \ in front of it and I've got a different error

Open in Slack

Previous Next