https://pinot.apache.org/ logo
j

Josh Highley

03/24/2021, 5:53 PM
When ingesting json data, do the json objects have to be flat? I can't find docs about parsing json object trees, but I see the github event example is a tree
m

Mayank

03/24/2021, 5:55 PM
It depends on what you are trying to do:
Copy code
1. If you want to store data as JSON blob in Pinot and use JSON indexing: <https://docs.pinot.apache.org/basics/indexing/json-index>

2. If you want to extract data from nested input JSON and store in a flat schema: 

<https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#column-transformation>

<https://docs.pinot.apache.org/users/user-guide-query/scalar-functions?q=groovy>
j

Josh Highley

03/24/2021, 5:57 PM
#2. The row data is several levels deep, and it's not an array but rather a list of id properties { node1: { node2: { node3: { id1: { col1: value1, col2: value2 }, id2: { col1: value1, col2: value2 } } } }
m

Mayank

03/24/2021, 5:58 PM
You can write a transform to extract that data and map to a column in Pinot schema
List can map to multi-valued columns
j

Josh Highley

03/24/2021, 6:00 PM
sorry, hit send before my example was finished. id1 is a row, id2 is a row, etc
m

Mayank

03/24/2021, 6:02 PM
You should still be able to write per column transform (would be a little inefficient to do it once per column)?
j

Josh Highley

03/24/2021, 6:14 PM
How does Pinot decide what the row level object is? ex: id1, id2, etc vs some other object
m

Mayank

03/24/2021, 6:14 PM
Your transform function would need to decide that