Hi team, I'm trying to ingest a stringified array ...
# troubleshooting
a
Hi team, I'm trying to ingest a stringified array field from Kafka into Apache Pinot and convert it into a proper multi-value column (
STRING[]
), but I'm facing segment consumption errors. Here’s what I’ve done so far: • Pinot version: 1.0+ • In Kafka, the message contains a field like: "txn_notification_event_list": "[\"ZORBLAT\",\"QUIXANO\",\"FLUMPTION\",\"WIBBLEX\",\"SNOOFLE-VORTEX\",\"PLINKO-FUND-SWAP\",\"ZOOGLE\",\"FX-TRANSMOG\"]" I tried the following
transformConfig
in the table config: Config 1 : { "columnName": "txn_notification_event_list_array", "transformFunction": "jsonExtractArray(txn_notification_event_list, '$', 'STRING')" } Config 2 : "transformConfigs": [ { "columnName": "txn_notification_event_list_array", "transformFunction": "jsonFormatArray(JSONPARSE(txn_notification_event_list))" } ] And defined this in the schema: { "name": "txn_notification_event_list_array", "dataType": "STRING", "singleValueField": false } • The segment ends up in ERROR state on both servers. No
segmentSize
, no
consumerInfo
, and no
errorInfo
shown in the Pinot UI. I've verified that the Kafka messages are correctly formatted as stringified arrays, but I’m not sure if Pinot is parsing this properly or if something’s misconfigured in my schema/table setup. Would appreciate any help or pointers! Thanks!
m
The upstream event doesn't look like a JSON to me, so JSON transforms wont work right?
You can try
stringToArray